Flexible accelerator for sparse tensors in convolutional neural networks

ABSTRACT

An apparatus includes a tensor compute cluster having a plurality of tensor compute units to process a plurality of sub-feature maps in a machine learning application and a tensor memory cluster having a plurality of tensor feature map memory units to store the plurality of sub-feature maps. The apparatus also includes circuitry to partition an input feature map into the plurality of sub-feature maps such that sparsity in each of the plurality of sub-feature maps satisfies a predetermined threshold, and assign each of the plurality of sub-feature maps to one of the plurality of tensor compute units and one of the plurality of tensor feature map memory units for processing in parallel.

BACKGROUND

Applicant provides the following description to assist the understandingof the reader. None of the information provided or references cited isadmitted to be prior art.

Machine learning algorithms are used in a wide variety of applicationssuch as medical imaging, computer vision, advertising, gaming, etc.Machine learning algorithms process a large amount of input data. As theamount of input data increases, the computation time and computationalresources needed to process the large amount of input data alsoincreases. However, currently used mechanisms are limited in the waythey are configured and the way they process input data in machinelearning applications.

SUMMARY

In accordance with some aspects of the present disclosure, an apparatusis disclosed. The apparatus includes a tensor compute cluster having aplurality of tensor compute units to process a plurality of sub-featuremaps in a machine learning application, a tensor memory cluster having aplurality of tensor feature map memory units to store the plurality ofsub-feature maps, and circuitry to partition an input feature map intothe plurality of sub-feature maps such that sparsity in each of theplurality of sub-feature maps satisfies a predetermined threshold. Thecircuitry also assigns each of the plurality of sub-feature maps to oneof the plurality of tensor compute units and one of the plurality oftensor feature map memory units for processing in parallel.

In accordance with some other aspects of the present disclosure, amethod is disclosed. The method includes partitioning, by an acceleratorin a machine learning application, an input feature map into a pluralityof sub-feature maps, such that sparsity in each of the plurality ofsub-feature maps satisfies a predetermined threshold. The method alsoincludes assigning, by the accelerator, each of the plurality ofsub-feature maps to one of a plurality of tensor compute units of atensor compute cluster for processing each of the plurality ofsub-feature maps in parallel, processing, by the accelerator, each ofthe plurality of sub-feature maps in the assigned one of the pluralityof tensor compute units to generate an output sub-feature map from eachof the plurality of tensor compute units, and combining, by theaccelerator, the output sub-feature map from each of the plurality oftensor compute units to generate an output feature map for the inputfeature map.

In accordance with yet other aspects of the present disclosure, anon-transitory computer-readable media having computer-readableinstructions stored thereon is disclosed. The computer-readableinstructions when executed by a processor associated with an acceleratorin a machine learning application cause the processor to partition aninput feature map into a plurality of sub-feature maps such that each ofthe plurality of sub-feature maps comprises substantially similarsparsity, and assign each of the plurality of sub-feature maps to asparse tensor compute cluster for processing each of the plurality ofsub-feature maps in parallel.

In accordance with some other aspects of the present disclosure, anotherapparatus is disclosed. The apparatus includes means for partitioning aninput feature map into a plurality of sub-feature maps based uponsparsity and means for assigning each of the plurality of sub-featuremaps to a tensor compute cluster for processing each of the plurality ofsub-feature maps in parallel.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the following drawings and thedetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example block diagram of a computing system, in accordancewith some embodiments of the present disclosure.

FIG. 2 is an example block diagram of an accelerator of the computingsystem of FIG. 1, in accordance with some embodiments of the presentdisclosure.

FIGS. 3A and 3B are examples for converting an input feature map into aplurality of sub-feature maps, in accordance with some embodiments ofthe present disclosure.

FIG. 4 is an example block diagram of a sparse tensor memory cluster ofthe accelerator of FIG. 2, in accordance with some embodiments of thepresent disclosure.

FIG. 5 is another example block diagram of the sparse tensor memorycluster of the accelerator of FIG. 2, in accordance with someembodiments of the present disclosure.

FIG. 6 is an example block diagram of a sparse tensor feature map memoryunit of the sparse tensor memory clusters of FIGS. 4 and 5, inaccordance with some embodiments of the present disclosure.

FIG. 7 is an example block diagram of a sparse tensor weight memory ofthe sparse tensor memory clusters of FIGS. 4 and 5, in accordance withsome embodiments of the present disclosure.

FIG. 8 is an example block diagram of a sparse tensor compute cluster ofthe accelerator of FIG. 2, in accordance with some embodiments of thepresent disclosure.

FIG. 9 is another example block diagram of the sparse tensor computecluster of the accelerator of FIG. 2, in accordance with someembodiments of the present disclosure.

FIG. 10 is an example block diagram of a sparse tensor compute unit ofthe sparse tensor compute clusters of FIGS. 8 and 9, in accordance withsome embodiments of the present disclosure.

FIG. 11 is an example of processing a sub-feature map in the sparsetensor compute unit of FIG. 10, in accordance with some embodiments ofthe present disclosure.

FIG. 12 is an example flowchart outlining operations for processing thesub-feature map in the sparse tensor compute unit of FIG. 10, inaccordance with some embodiments of the present disclosure.

FIG. 13 is another example of the sparse tensor compute unit of thesparse tensor compute clusters of FIGS. 8 and 9, in accordance with someembodiments of the present disclosure.

FIG. 14 is yet another example of the sparse tensor compute unit of thesparse tensor compute clusters of FIGS. 8 and 9, in accordance with someembodiments of the present disclosure.

FIG. 15 is one more example of the sparse tensor compute unit of thesparse tensor compute clusters of FIGS. 8 and 9, in accordance with someembodiments of the present disclosure.

FIG. 16 is an example flowchart outlining operations for processing asub-feature map in the sparse tensor compute units of FIGS. 13-15 in astandard convolution operation or in a 1×1 fully connected convolutionoperation, in accordance with some embodiments of the presentdisclosure.

FIG. 17 is an example flowchart outlining operations for processing aninput feature map using the accelerator of FIG. 2, in accordance withsome embodiments of the present disclosure.

FIGS. 18A-18D show an example of processing a sub-feature map in thesparse tensor compute unit of FIG. 13, in accordance with someembodiments of the present disclosure.

FIGS. 19A-19E show an example of processing a sub-feature map in thesparse tensor compute unit of FIG. 15, in accordance with someembodiments of the present disclosure.

FIGS. 20A-20B show an example of a merging operation, in accordance withsome embodiments of the present disclosure.

The foregoing and other features of the present disclosure will becomeapparent from the following description and appended claims, taken inconjunction with the accompanying drawings. Understanding that thesedrawings depict only several embodiments in accordance with thedisclosure and are therefore, not to be considered limiting of itsscope, the disclosure will be described with additional specificity anddetail through use of the accompanying drawings.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe figures, can be arranged, substituted, combined, and designed in awide variety of different configurations, all of which are explicitlycontemplated and made part of this disclosure.

Neural networks keep advancing towards larger and deeper architecturesas they get involved in more sophisticated processing tasks in a broaderscope of scenarios. Large neural networks intensify the computation andmemory accesses with increasing amount of data. Specifically, the numberof synapses and neurons in a neural network increase each time theamount of data that needs to be processed increases. For example, someneural networks may have over six hundred thousand neurons, while someother neural networks may have more than one million neurons. The numberof synapses may be much larger, for example, over sixty million in oneneural network. Convolutional Neural Networks (“CNN”) are one type ofneural networks. Specifically, CNN is a type of a deep neural networkthat uses convolution operations to classify an input (e.g., an image).To classify, the CNN receives the input and assigns weights and biasesto the input to differentiate one aspect of the input from anotheraspect of the input. In some embodiments, CNN may include multiplelayers of processing between receiving the input and sending the output.Depending upon the size of the input and the number of layers used inthe CNN, a large amount of data may need to be processed.

Processing such large amounts of data brings challenges for off-chip andon-chip memory bandwidth in accelerator architecture design. Mechanismsmay be used to either reduce the amount of data on which computation isto be performed or reduce the computation time. One such mechanism thatreduces the amount of data to be computed or the computation time mayinclude weight encoding via quantization and entropy coding that shrinkthe weight size. Other mechanisms work at the circuit-level, such asusing shorter bit-width operators or approximate computing. Anothermechanism to reduce computation time or amount of data relies onsparsity.

Sparsity may determine the number or percentage of non-zeroes in a givendata. Pruning synapses and neurons in a neural network based on sparsitymay reduce the amount of data that needs to be computed by more than tentimes with negligible accuracy loss. Sparsity may be either static ordynamic. Static sparsity is known beforehand and may not change from oneset of input data to another set of input data. The sparsity in weightsmay be considered static sparsity. For example, the weights that arezero or non-zero in value may be known before the computation on theinput data is begun. Further, the weights that are zero or non-zero mayremain the same from one set of input data to the next. Dynamic sparsityis the sparsity in an input feature map of the input data.

Specifically, from an input data (e.g., an input image), one or moreinput feature maps may be generated. Each of the input feature maps maybe combined with associated weights to perform a classification process.Each input feature map may have either zero values or non-zero values.The number or percentage of non-zero values in a given input feature mapmay determine the sparsity of that input feature map. Since each inputfeature map may be different from another input feature map, and thelocation of the zero and non-zero values may change from one inputfeature map to another, the sparsity in an input feature map is dynamicsparsity. Since static sparsity is easier to identify and consider in amachine learning operation, many mechanisms that reduce the amount ofdata to be computed rely on static sparsity only. The irregularitycaused by dynamic sparsity prevents many mechanisms from fullyleveraging the computation and data reduction.

Thus, the present disclosure provides mechanisms to transform dynamicand random sparsity into a more or less structured sparsity for reducingboth the amount of data that is to be computed, as well as for reducingthe computation time. Specifically, the present disclosure provides aflexible accelerator that converts an input feature map into a pluralityof input sub-feature maps such that each of the plurality of sub-featuremaps has a similar amount of sparsity and allows each of the pluralityof sub-feature maps to be processed independently while taking advantageof the sparsity.

Referring now to FIG. 1, an example block diagram of a computing system100 is shown, in accordance with some embodiments of the disclosure. Thecomputing system 100 may include a host device 105 associated with amemory device 110. The host device 105 may be configured to receiveinput from one or more input devices 115 and provide output to one ormore output devices 120. The host device 105 may be configured tocommunicate with the memory device 110, the input devices 115, and theoutput devices 120 via appropriate interfaces 125A, 125B, and 125C,respectively. The computing system 100 may be implemented in a varietyof computing devices such as computers (e.g., desktop, laptop, servers,data centers, etc.), tablets, personal digital assistants, mobiledevices, wearable computing devices such as smart watches, otherhandheld or portable devices, or any other computing unit suitable forperforming operations using the host device 105.

The input devices 115 may include any of a variety of input technologiessuch as a keyboard, stylus, touch screen, mouse, track ball, keypad,microphone, voice recognition, motion recognition, remote controllers,input ports, one or more buttons, dials, joysticks, and any other inputperipheral that is associated with the host device 105 and that allowsan external source, such as a user, to enter information (e.g., data)into the host device and send instructions to the host device.Similarly, the output devices 120 may include a variety of outputtechnologies such as external memories, printers, speakers, displays,microphones, light emitting diodes, headphones, plotters, speechgenerating devices, video devices, global positioning systems, and anyother output peripherals that are configured to receive information(e.g., data) from the host device 105. The “data” that is either inputinto the host device 105 and/or output from the host device may includeany of a variety of textual data, graphical data, video data, imagedata, sound data, position data, combinations thereof, or other types ofanalog and/or digital data that is suitable for processing using thecomputing system 100.

The host device 105 may include one or more Central Processing Unit(“CPU”) cores or processors 130A-130N that may be configured to executeinstructions for running one or more applications associated with thehost device. The CPU cores 130A-130N are shown as a non-limitingrepresentative example of integrated circuits that can performprocessing functions, and may be substituted and/or combined with FieldProgrammable Gate Array (“FPGA”), Graphical Processing Unit (“GPU”),custom Application Specific Integrated Circuit (“ASIC”), and the like.In some embodiments, the instructions and data needed to run the one ormore applications may be stored within the memory device 110. The hostdevice 105 may also be configured to store the results of running theone or more applications within the memory device 110. The host device105 may also include an accelerator 135. The accelerator 135 may be usedto perform machine learning operations. The accelerator 135 is discussedin greater detail in FIG. 2. Although the accelerator 135 is shown asbeing part of the host device 105 in FIG. 1, in other embodiments, theaccelerator may be apart from the host device and communicativelycoupled (e.g., through a bus or network connection) to the host device.In such a case, the accelerator 135 may also be communicatively coupledto the memory device 110, be a part of the memory device 110, or includeits own separate memory device.

To facilitate communication with the memory device 110, the memorydevice may include or be associated with a memory controller 140.Although the memory controller 140 is shown as being part of the memorydevice 110, in some embodiments, the memory controller may instead bepart of the host device 105 or another element of the computing system100 and operatively associated with the memory device. The memorycontroller 140 may be configured as a logical block or circuitry thatreceives instructions from the host device 105 (e.g., the accelerator135) and performs operations in accordance with those instructions. Thememory device 110 may include one or more memory modules 145 that storedata and instructions. The memory modules 145 may be any of a variety ofmemory types, including a variety of volatile memories, non-volatilememories, or a combination thereof. For example, in some embodiments,one or more of the memory modules 145 or portions thereof may includeNAND flash memory cores. In other embodiments, one or more of the memorymodules 145 or portions thereof may include NOR flash memory cores,Static Random Access Memory (SRAM) cores, Dynamic Random Access Memory(DRAM) cores, Magnetoresistive Random Access Memory (MRAM) cores, PhaseChange Memory (PCM) cores, Resistive Random Access Memory (ReRAM) cores,3D XPoint memory cores, ferroelectric random-access memory (FeRAM)cores, and other types of memory cores that are suitable for use withinthe memory device 110. In some embodiments, one or more of the memorymodules 145 or portions thereof may be configured as other types ofstorage class memory (“SCM”). Generally speaking, the memory modules 145may include any of a variety of Random Access Memory (RAM), Read-OnlyMemory (ROM), Programmable ROM (PROM), Erasable PROM (EPROM),Electrically EPROM (EEPROM), hard disk drives, flash drives, memorytapes, cloud memory, or any combination of primary and/or secondarymemory that is suitable for performing the operations described herein.

It is to be understood that only some components of the computing system100 are shown and described in FIG. 1. However, the computing system 100may include other components such as various batteries and powersources, networking interfaces, routers, switches, external memorysystems, controllers, etc. Generally speaking, the computing system 100may include any of a variety of hardware, software, and/or firmwarecomponents that are needed or considered desirable in performing thefunctions described herein. Similarly, the host device 105, the inputdevices 115, the output devices 120, and the memory device 110 includingthe accelerator 135, the memory controller 140, and the memory modules145 may include other hardware, software, and/or firmware componentsthat are considered necessary or desirable in performing the functionsdescribed herein. In addition, in certain embodiments, the memory device110 may integrate some or all of the components of the host device,including, for example, the CPU cores 130A-130N and/or the accelerator135.

Turning now to FIG. 2, an example accelerator 200 is shown, inaccordance with some embodiments of the present disclosure. Theaccelerator 200 is analogous to the accelerator 135 of FIG. 1. Thus,although not shown, the accelerator 200 may be associated with a hostdevice (e.g., the host device 105) and a memory device (e.g., the memorydevice 110). The accelerator 200 may be used to optimize machinelearning operations, for example, in a CNN. Although the accelerator 200is explained with respect to CNN, in other embodiments, the accelerator200 may be used in other types of neural networks or machine learningapplications as well. Generally speaking, the accelerator 200 may beused in any type of application (whether machine learning or otherwise)that desires to reduce computation data and computation time.

In some embodiments, the accelerator 200 may be used to consider dynamicand static sparsity in the input feature maps and the weights,respectively, and allocate computation amongst various compute enginesbased on the dynamic and static sparsity. The accelerator 200 mayreceive an input image 205 (e.g., from the host device 105) on which oneor more machine learning operations are to be performed. It is to beunderstood that although the input image 205 is used herein as anexample, the accelerator 200 may be used to process other types of dataincluding video, text, and any other type of data that may benefit frombeing processed by the accelerator.

The input image 205 may be represented by an array of pixels. Simply asan example and without intending to be limited in any way, say the inputimage 205 is represented by a 1024×1024×3 array of pixels. Thus, theinput image 205 is 1024 pixels high, 1024 pixels wide, and 3 colors(e.g., Red, Green, Blue) deep. In some embodiments, the 1024×1024×3array of pixels may be divided into three input feature maps, with eachinput feature map representing one color and being of size 1024×1024×1(also referred to herein as simply 1024×1024). Further, each inputfeature map may be represented as a matrix having a plurality of rowsand a plurality of columns. Each row extends in an X-direction(left-right), while each column extends in a Y-direction (up-down). Eachpixel of an input feature map may correspond to one cell (e.g., formedat the intersection of one row and one column) of the matrix. Thus, a1024×1024 input feature map may be represented by a matrix having 1024rows and 1024 columns, with the intersection of each row and each columnforming one cell for one pixel.

In some embodiments, the input feature maps of the input image 205 maybe generated by the accelerator 200. For example, in some embodiments, apartitioning block 210 may be configured to generate the input featuremaps from the input image 205. In other embodiments, a feature map block(not shown) of the accelerator 200 may receive the input image 205, andgenerate the input feature maps therefrom, and send those input featuremaps to the partitioning block 210. In yet other embodiments, the inputfeature maps of the input image 205 may be generated outside of theaccelerator 200, and instead of the input image, the input feature mapsmay be input into the accelerator. The partitioning block 210 may beconfigured in software, hardware, firmware, or combinations thereof.

Each of the input feature maps of the input image 205 may be furthersub-divided into one or more sub-feature maps in the partitioning block210. The sub-division of an input feature map into one or moresub-feature maps in the partitioning block 210 is discussed in FIGS. 3Aand 3B below. Each sub-feature map is also referred to herein as an“input tensor.” By dividing an input feature map into one or moresub-feature maps, each of the sub-feature maps may be independentlyprocessed to generate an output sub-feature map and to increaseperformance. The output sub-feature maps from each of the sub-featuremaps may then be combined together to create an output feature map. Whenthe image 205 includes multiple channels, the input feature maps of eachchannel may be sub-divided into one or more sub-feature maps, each ofthe sub-feature maps may be processed independently to generate anoutput sub-feature map, and the output sub-feature maps of each of theone or more sub-feature maps may be combined to generate a channeloutput feature map. The various channel output feature maps may then becombined to generate the output feature map. Such convolution operationsthat involves multiple channels is referred to herein as “depth wiseseparable convolution.” In some embodiments, the input feature mapsand/or each of the sub-feature maps may be compressed to reduce storagespace and further increase performance. The process of compressing asused throughout this disclosure is discussed in greater detail in U.S.application Ser. No. 16/726,084, titled “Flexible Accelerator For SparseTensors (FAST) in Machine Learning” and filed on Dec. 23, 2019, theentirety of which is incorporated by reference herein.

The partitioning block 210 may be associated with a DRAM 215 that may beconfigured to initially store the input feature maps of the input image205, and upon sub-dividing the input feature maps into one or moresub-feature maps, store each of the one or more sub-feature maps. TheDRAM 215 may also be used to store any intermediate outputs (e.g., theoutput sub-feature map, channel output feature map, etc.) and/or theoutput feature map. In some embodiments, the DRAM 215 may also beconfigured to store a weight matrix 220. In some embodiments, the DRAM215 may also store various training models, schedules, and otherinformation needed to process the input image 205. Further, althoughonly the DRAM 215 is shown in the accelerator 200, in some embodiments,the accelerator may include additional and/or other types of memories.For the sake of simplicity, the DRAM 215 is used in the description, butother memory substitutes are contemplated for various embodiments. Forexample, in some embodiments, the accelerator 200 may additionally oralternatively include SRAM, storage class memory such as MRAM, ReRAM,and/or flash memories to store the input feature maps, the one or moresub-feature maps, various outputs, training models, schedules, and/orother information needed to process the input image 205. When suchmultiple memories are provided in the accelerator 200, in someembodiments, these memories may be interconnected with each other andconfigured to communicate with each other. In some embodiments, the DRAM215 and any other memory in the accelerator 200 may be part of thememory device (e.g., the memory device 110) associated with the hostdevice (e.g., the host device 105) of which the accelerator is part ofIn some embodiments, one or more of the DRAM 215 and any other memory inthe accelerator 200 may be separate from the memory device (e.g., thememory device 110).

In addition to the input image 205, the accelerator 200 also receivesthe weight matrix 220 (e.g., from the host device 105). The weightmatrix 220 may include weights or filters that are to be applied to eachof the sub-feature maps. The weight matrix 220 is also referred toherein as a kernel matrix or a filter matrix. The weight matrix 220 mayinclude at least one row and at least one column, forming cells at theintersection of rows and columns. The weight matrix 220 may be used toperform the convolution operations on the input image 205. In someembodiments, the weight matrix 220 may also be sub-divided into one ormore sub-weight maps similar to the input feature maps. In someembodiments, the weight matrix 220 and/or the sub-weight maps may alsobe compressed similar to the input feature maps/sub-feature maps. Insome embodiments, the weight matrix 220 may be received by a schedulingengine 225. In other embodiments, the weight matric 220 may be receivedby the partitioning block 210 or by another component of the accelerator200. Further, if the weight matrix 220 is sub-divided into sub-weightmaps and/or compressed, these processes may occur within the schedulingengine 225 in some embodiments. In other embodiments, these processesmay occur in the portioning block 210, in a separate partitioning blockin the accelerator 200 dedicated to the weight matrix 220, and/oroutside the accelerator. The weight matrix 220 and/or the sub-weightmaps may be stored within the DRAM 215, the scheduling engine 225, or inany other memory of the accelerator 200.

The scheduling engine 225 is configured to perform a sparsity analysis,and assign, in some embodiments, each of the input feature maps to acompute unit based upon the sparsity. As used herein, “sparsity” meansthe number or percentage of non-zeroes in a given input data. In someembodiments, an input feature map that has more non-zeroes than zeroesis a less sparse input feature map or a dense input feature map,whereas, an input feature map that has more zeroes than non-zeroes is asparse input feature map. In other embodiments, a sparse input featuremap may be defined as having at least a certain percentage of zeros(e.g., 80%), and a dense input feature map may have more zeroes thannon-zeroes (e.g., 60:40). More generally, a sparse input feature map hasmore zeros than a dense input feature map. Thus, to perform the sparsityanalysis, the scheduling engine 225 may determine the number orpercentage of zeroes or non-zeroes in an input feature map. Further, thescheduling engine 225 may assign or allocate a sparse input feature mapto a sparse tensor compute cluster 230 for processing. Thus, the sparsetensor compute cluster 230 is configured to process input feature mapshaving higher sparsity (e.g., number or percentage of zeroes above athreshold, or in other words, more zero values than non-zero values).The sparse tensor compute cluster 230 is discussed in greater detailbelow.

In some embodiments, the accelerator 200 may also include a dense tensorcompute cluster (not shown in FIG. 2) and/or a vector accelerator (alsonot shown in FIG. 2). If provided, the dense tensor compute cluster maybe configured to process feature maps having lower sparsity (e.g., denseinput feature maps or input feature maps in which the number orpercentage of zeroes is below a threshold, or in other words, havingmore non-zero values than zero values) and the vector accelerator may beused to process input feature maps that cannot be processed by eitherthe dense tensor compute cluster or the sparse tensor compute cluster230. For example, in some embodiments where the dense tensor computecluster and/or a vector accelerator are provided, the dense tensorcompute cluster and the sparse tensor compute cluster 230 may beconfigured with a defined or supported list of operations that the densetensor compute cluster and the sparse tensor compute cluster,respectively, may be able to perform. If the processing requiresperforming an operation that is not supported by either of the densetensor compute cluster or the sparse tensor compute cluster 230, thenthat operation may be performed by the vector accelerator. In someembodiments, the vector accelerator may be an open source vectoraccelerator based on RISC-V such as LACore. In other embodiments, othertypes of suitable vector accelerators may be used for the vectoraccelerator. The dense tensor compute cluster and the vector acceleratorare discussed in greater detail in the U.S. application Ser. No.16/726,084 mentioned above, again the entirety of which is incorporatedby reference herein.

The scheduling engine 225 may include a memory 235 to storecomputer-readable instructions that may be executed by a processor 240to perform the operations of the scheduling engine 225. The memory 235may be part of the memory device (e.g., the memory device 110) of thehost device (e.g., the host device 105) of which the accelerator 200 ispart of, or may be provisioned from a separate memory. Further, thescheduling engine 225 may be implemented as software, hardware,firmware, or combination thereof. The processor 240 may be part of ormay be similar to the CPU cores 130A-130N of the host device 105, or inthe case when the accelerator is part of the memory device 110, part ofa processor or controller of the memory device.

For processing an input feature map in the sparse tensor compute cluster230, the accelerator 200 may also include a direct memory accesscontroller 245 configured to transfer the sub-feature maps from the DRAM215 to a sparse tensor memory cluster 250 with minimal intervention fromthe scheduling engine 225 or the sparse tensor compute cluster. Thesparse tensor memory cluster 250 is also discussed in greater detailbelow. It is noted while direct memory access is used as an example forillustration, other type of memory access protocols/methods may be used,including memory access across communication buses and memory fabrics.

It is to be understood that only some components of the accelerator 200are shown in FIG. 2. The accelerator 200 may include other or additionalcomponents that may be needed or considered desirable in performing theoperations described herein.

Turning to FIGS. 3A and 3B, an example of dividing an input feature mapinto one or more sub-feature maps or input tensors in the partitioningblock 210 is shown, in accordance with some embodiments of the presentdisclosure. Specifically, an input feature map that is a sparse inputfeature map and has been designated to be processed in the sparse tensorcompute cluster 230 may be sub-divided into one or more sub-featuremaps. Specifically, each input feature map for each channel may bedivided into “M” sub-feature maps. “M” is dependent upon a number ofsparse tensor compute units provided within the sparse tensor computecluster 230 and/or the number of parse tensor feature map memory unitsin the sparse tensor memory cluster 250. For example, for five sparsetensor compute units in the sparse tensor compute cluster 230, eachinput feature map of each channel may be divided into five sub-featuremaps. By dividing each input feature map into “M” sub-feature mapscorresponding to “M” number of sparse tensor compute units in the sparsetensor compute cluster 230, each of the “M” sub-feature maps may beprocessed independently in one of the “M” sparse tensor compute units toincrease performance and reduce computation time.

In some embodiments, each of the “M” sub-feature maps may be a matrixhave overlapping (k−1, as an example) rows and (k−1, as an example)columns with neighboring sub-feature maps, where (k×k) is the size ofthe weight matrix 220. By overlapping rows and columns of onesub-feature map with a neighboring sub-feature map, each sub-feature mapmay be processed independently in the sparse tensor compute cluster 230and the output sub-feature maps of each of the sub-feature maps may becombined together without needing any intra-cluster communicationbetween the “M” sub-feature maps to create an output feature map. Inother embodiments, the “M” sub-feature maps may have no overlapping rowsand/or columns. Without overlapping, the intermediate results fromneighboring sub-feature maps may need to be combined. In someembodiments, with a kernel size of 1×1, overlapping rows and/or columnsof the neighboring sub-feature maps may not be needed.

Each input feature map may be divided into “M” sub-feature maps based onsparsity, and specifically, to convert the dynamic nature of sparsity inthe input feature map into close to static sparsity. Simply as anexample and without intending to be limiting in any way, say the inputfeature map of a particular channel of the input image 205 is 512×512 insize (e.g., 512 rows and 512 columns), and the number of sparse tensorcompute units in the sparse tensor compute cluster 230 is 16 (in otherwords, “M”=16). Thus, the 512×512 input feature map may be divided into16 sub-feature maps, with each sub-feature map being assigned to one ofthe 16 sparse tensor compute units of the sparse tensor compute cluster230. To create the 16 sub-feature maps, the 512×512 input feature mapmay be initially divided into 16 equal sized or roughly equal sizedpartitions, such that each partition has a number of rows and a numberof columns.

For example, FIG. 3A shows an example table 300 after dividing the512×512 input feature map into 16 sub-feature maps (e.g., generallyrepresented by cell 1-cell 16). Thus, each of the 16 partitions of the512×512 input feature map may be 32×32 (e.g., 32 rows and 32 columns) insize. Each cell of the table 300 corresponds to one 32×32 partition ofthe input feature map. The value in the parenthesis in each cell in FIG.3A indicates the percentage of non-zeroes in that particular cell. Thus,for example, cell 1 of the table 300 corresponds to the first partition(e.g., rows 1 to 32 and columns 1 to 32 of the input feature map) andincludes 40% non-zeroes (and therefore, 60% zeroes), cell 2 correspondsto the second partition (e.g., rows 1 to 32 and columns 33-64) andincludes 60% non-zeroes (and therefore, 40% zeroes), and so on. Cell 5corresponds to the fifth partition (rows 33 to 64 and columns 1 to 32 ofthe input feature map) and includes 51% non-zeroes, and so on. In theexample of FIG. 3A, there is no overlap in the rows and columns of the16 sub-feature maps. However, as discussed above, in some embodiments,at least some of the sub-feature maps may have overlapping rows and/orcolumns with neighboring sub-feature maps. In such cases, the number ofthe sub-feature maps may vary from 16 in the example of FIG. 3A.

The initial partitions of the table 300 may be reorganized such thateach partition includes the same or substantially similar number orpercentage of non-zeroes and/or zeroes. In some embodiments, apercentage difference of non-zeroes (or zeroes) may be pre-determinedsuch that any two given partitions may not have a greater than thepre-determined percentage difference in the non-zeroes. As an example,if the pre-determined percentage difference in the percentage ofnon-zeroes between any two given partitions is 3%, the initialpartitions of FIG. 3A may be reorganized such that each partition hasroughly equal number of non-zeroes, with no two partitions havinggreater than a 3% difference in the number of non-zeroes. Thus,referring to FIG. 3B, another table 305 is shown that shows how thecells from the table 300 are reorganized. For example, cell 1 in thetable 300 has 40% non-zeroes, while cell 2 in the table 300 has 60%non-zeroes. Thus, the percentage difference in the non-zeroes betweencell 1 and cell 2 in table 300 is greater than the pre-determinedpercentage difference of 3%. Thus, cells 1 and 2 may be reorganized asshown in the table 305 by allocating columns from cell 2 to cell 1. Forexample, three columns from cell 2 may be allocated to cell 1 such thateach of cell 1 and cell 2 has about 50% non-zeroes, thereby equalizingthe number of non-zeroes in those cells.

After the reorganization, cell 1 may have 32 rows and 35 columns, and istherefore 32×35 in size, as shown in the table 305 instead of the 32×32size in the table 300. Similarly, after the organization, cell 2 mayhave 32 rows and 29 columns, as shown in the table 305. Thus, cells 1and 2 are of different sizes, as shown in the table 305. Although cell 1is allocated 3 columns from only one neighboring cell, in someembodiments, cell 1 may be allocated rows/columns from multipleneighboring cells. For example, in some embodiments, cell 1 may beallocated 2 columns from cell 2 and 1 row from cell 5. The number ofrows and/or the number of columns that are allocated from one or morecells to a particular cell may be dependent upon the number ofnon-zeroes (or zeroes) that are present in a particular row and/orcolumn, and the number of non-zeroes (or zeroes) that are needed to bereallocated to satisfy the pre-determined percentage difference.Likewise, the other cells in the table 300 may be balanced out byallocating one or more rows and/or one or more columns from one or moreneighboring cells until the pre-determined difference is satisfied, asshown in the table 305.

By reorganizing the initial partitions shown in the table 300, thenumber of non-zeroes in the various initial partitions may be balancedto balance out the processing in the sparse tensor compute cluster 230.Each cell in the table 305 corresponds to one sub-feature map or oneinput tensor, which may be assigned to one sparse tensor feature mapmemory unit of the sparse tensor memory cluster 250 and one sparsetensor compute unit of the sparse tensor compute cluster 230. AlthoughFIGS. 3A and 3B have been explained as going from the initial partitionof the table 300 to the final partition of the table 305 in a singlestep, in other embodiments, multiple iterations of reallocatingrows/columns to/from neighboring cells may be needed to achieve thepre-determined percentage difference. Further, although FIGS. 3A and 3Bhave been explained with respect to the pre-determined percentagedifference of non-zeroes, in other embodiments, different metrics may beused. For example, in some embodiments, instead of the percentage ofnon-zeroes, a percentage of zeroes may be used. Similarly, in someembodiments, instead of the percentage, a number of zeroes or non-zeroesmay be used. In yet other embodiments, metrics other than “number” maybe used. Each of the sub-feature maps of FIG. 3B may be stored withinthe DRAM 215.

Referring to FIG. 4, an example of a sparse tensor memory cluster 400 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor memory cluster 400 is analogous to the sparse tensormemory cluster 250 of FIG. 2. The sparse tensor memory cluster 400includes a sparse tensor feature map memory 405 and a sparse tensorweight memory 410. The sparse tensor feature map memory 405 isconfigured to store the sub-feature maps received from the DRAM 215 andthe direct memory access controller 245. The sparse tensor feature mapmemory 405 is also configured to store the various outputs received fromthe sparse tensor compute cluster 230. The sparse tensor feature mapmemory 405 includes a plurality of sparse tensor feature map memoryunits 415A-415M. The number of the plurality of sparse tensor featuremap memory units 415A-415M may be dependent upon a designated number ofsub-feature maps that may be desired to be processed in parallel. Eachof the plurality of sparse tensor feature map memory units 415A-415M isindependent from other sparse tensor feature map memory units and may beconfigured to store at least one sub-feature map independent from othersparse tensor feature map memory units.

Thus, in some embodiments, the plurality of sparse tensor feature mapmemory units 415A-415M are not configured to share the data storedtherein with other ones of the plurality of sparse tensor feature mapmemory units. Further, each of the plurality of sparse tensor featuremap memory units 415A-415M is configured to send the sub-feature mapstored therein to a corresponding one of a sparse tensor compute unit ofthe sparse tensor compute cluster (e.g., the sparse tensor computecluster 230). For example, in some embodiments, the sparse tensorfeature map memory unit #i may be configured to send the input tensorstored therein to the sparse tensor compute unit #i, as discussedfurther below. Such one-to-one correspondence between a particularsparse tensor feature map memory unit and a sparse tensor compute unitis referred to herein as “static binding.” Thus, in some embodiments,the number of the plurality of sparse tensor feature map memory units415A-415M in the sparse tensor feature map memory 405 is same as thenumber of sparse tensor compute units in the sparse tensor computecluster (e.g., the sparse tensor compute cluster 230).

Further, each of the plurality of sparse tensor feature map memory units415A-415M may be connected via a bi-directional bus 420A-420M,respectively, to receive sub-feature maps from the DRAM 215 via thedirect memory access controller 245, as well as to send outputs receivedfrom the sparse tensor compute cluster 230 back to the DRAM via thedirect memory access controller. Similarly, each of the plurality ofsparse tensor feature map memory units 415A-415M may be connected via abi-directional bus 425A-425M, respectively, to an associated one of thesparse tensor compute unit of the sparse tensor compute cluster (e.g.,the sparse tensor compute cluster 230) to send the sub-feature mapsstored therein to the sparse tensor compute cluster and to receiveoutputs back from the sparse tensor compute cluster.

Thus, for example, the sparse tensor feature map memory unit 415A mayreceive a sub-feature map from the DRAM 215 via the direct memory accesscontroller 245 and the bus 420A for storing, and send that sub-featuremap to an associated one of the sparse tensor compute unit of the sparsetensor compute cluster (e.g., the sparse tensor compute cluster 230) forprocessing via the bus 425A. Similarly, the sparse tensor feature mapmemory unit 415A may receive the output (e.g., the result fromprocessing the sub-feature map) from the sparse tensor compute cluster(e.g., the sparse tensor compute cluster 230) via the bus 425A forstoring, and send that output to the DRAM 215 via the direct memoryaccess controller 245 and the bus 420A. The sparse tensor feature mapmemory units 415B-415M may function similar to the sparse tensor featuremap memory unit 415A.

In some embodiments, each of the plurality of sparse tensor feature mapmemory units 415A-415M may also be configured to store index values ofthe sub-feature map that is stored therein. In addition to receiving asub-feature map, each of the plurality of sparse tensor feature mapmemory units 415A-415M may also receive the index values associated withthe sub-feature map from the DRAM 215. For example, if the sparse tensorfeature map memory units 415A receives sub-feature map A from the DRAM215, that sparse tensor feature map memory unit may also receive theindex values corresponding to the sub-feature map A. The sparse tensorfeature map memory units 415A may then send the index values of thesub-feature map A to the sparse tensor compute cluster (e.g., the sparsetensor compute cluster 230) along with sending the sub-feature map A.The index values capture the row numbers and column numbers of aparticular sub-feature map in the input feature map. For example, anindex value (X, Y) refers to the row number X and column number Y of thesub-feature map in the input feature map.

The sparse tensor weight memory 410 may be configured to store theweights that are to be applied to the sub-feature maps stored within thesparse tensor feature map memory units 415A-415M. Thus, the sparsetensor weight memory 410 may be connected via a uni-directional bus 430to the DRAM 215 and the direct memory access controller 245 to receivethe weights and via a bus 435 to the sparse tensor compute cluster(e.g., the sparse tensor compute cluster 230) for sending the weights tothe sparse tensor compute cluster. Since the sparse tensor weight memory410 does not need to receive any results back from the sparse tensorcompute cluster and does not need to send any results back to the DRAM215, the bus 430 and the bus 435 may be uni-directional buses configuredto send data in a single direction. In other embodiments, the bus 430and/or the bus 435 may be bi-directional similar to the bi-directionalbus 420A-420M/the bi-directional bus 425A-425M.

Turning to FIG. 5, an example of a sparse tensor memory cluster 500 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor memory cluster 500 is analogous to the sparse tensormemory cluster 250 of FIG. 2. The sparse tensor memory cluster 500 isalso substantially similar to the sparse tensor memory cluster 400. Forexample, similar to the sparse tensor memory cluster 400, the sparsetensor memory cluster 500 includes a sparse tensor feature map memory505 and a sparse tensor weight memory 510. Also similar to the sparsetensor feature map memory 405, the sparse tensor feature map memory 505includes a plurality of sparse tensor feature map memory units 515A-515Mconnected via a bi-directional bus 520A-520M to the DRAM 215 and thedirect memory access controller 245. However, unlike the sparse tensorfeature map memory 405 in which each of the plurality of sparse tensorfeature map memory units 515A-515M is independent, does not share thedata stored therein with other ones of the plurality of sparse tensorfeature map memory units, and sends the data stored therein to thecorresponding one of the sparse tensor compute unit, the plurality ofsparse tensor feature map memory units 515A-515M of the sparse tensorfeature map memory 505 are interconnected to one another and to othersparse tensor compute units via a memory interconnect 525.

Further, in some embodiments, the memory interconnect 525 may beconfigured to override the static binding discussed above. For example,in some embodiments, the memory interconnect 525 may enable a sparsetensor feature map memory unit #i to communicate with sparse tensorcompute unit #1−M (“M” is the number of the sparse tensor compute unitsin the associated sparse tensor compute cluster) depending upon theconfiguration of the memory interconnect. In some embodiments, thememory interconnect 525 may be two-by-two switch that enables a sparsetensor feature map memory unit #i to communicate with the sparse tensorcompute unit #i or sparse tensor compute unit #i+1. In otherembodiments, the memory interconnect 525 may be a multi-stageinterconnect such as a mesh network or Benes Network that allows asparse tensor feature map memory unit #i to communicate with each of thesparse tensor compute units #1−M. In yet other embodiments, the memoryinterconnect 525 may be configured in other ways to allow a sparsetensor feature map memory unit #i to communicate with one or more sparsetensor compute units in addition to the sparse tensor compute unit #i.Similarly, in some embodiments, the memory interconnect 525 may enable aparticular one of the plurality of sparse tensor feature map memoryunits 515A-515M to be interconnected with one or more of the other onesof the plurality of sparse tensor feature map memory units. For example,depending upon the configuration of the memory interconnect 525, asparse tensor feature map memory unit #i may be interconnected with oneor more of the sparse tensor feature map memory units #(i+1)−M.

Each of the plurality of sparse tensor feature map memory units515A-515M may be connected to the memory interconnect 525 via abi-directional bus 530A-530M. Thus, each of the plurality of sparsetensor feature map memory units 515A-515M may be configured to send thesub-feature map (and corresponding index values) stored therein to thememory interconnect 525 and receive a sub-feature map (e.g., that isstored in another sparse tensor feature map memory unit) or an outputfrom the memory interconnect via their respective one of thebi-directional bus 530A-530M. Similarly, the memory interconnect 525 maybe connected to the sparse tensor compute cluster (e.g., the sparsetensor compute cluster 230) via a bi-directional bus 535A-535M to sendsub-feature maps (and the index values) to and receive outputs from thesparse tensor compute cluster. By using the memory interconnect 525, theflexibility in storing information within the plurality of sparse tensorfeature map memory units 515A-515M may be increased and the staticbinding of the sparse tensor memory cluster 400 may be overridden.

The sparse tensor weight memory 510 is similarly configured as thesparse tensor weight memory 410. Thus, the sparse tensor weight memory510 may be configured to store the weights that are to be applied to thesub-feature maps stored within the sparse tensor feature map memoryunits 515A-515M. Further, the sparse tensor weight memory 510 may beconnected via a uni-directional bus 540 to the DRAM 215 and the directmemory access controller 245 to receive the weights and via a bus 545 tothe sparse tensor compute cluster (e.g., the sparse tensor computecluster 230) for sending the weights to the sparse tensor computecluster. In other embodiments, the bus 540 and/or the bus 545 may bebi-directional.

Referring to FIG. 6, an example sparse tensor feature map memory unit600 is shown, in accordance with some embodiments of the presentdisclosure. The sparse tensor feature map memory unit 600 is analogousto each of the plurality of sparse tensor feature map memory units415A-415M and the plurality of sparse tensor feature map memory units515A-515M. The sparse tensor feature map memory unit 600 includes awrite switch 605, a read switch 610, a first set of buffers 615, and asecond set of buffers 620. The write switch 605 is configured to writethe sub-feature maps received from the DRAM 215 (or from another sparsetensor feature map memory unit if interconnected) to the first set ofbuffers 615 and/or the second set of buffers 620. The write switch 605is also configured to write the outputs (e.g., the output sub-featuremaps) received from the sparse tensor compute cluster (e.g., the sparsetensor compute cluster 230) to the first set of buffers 615 and/or thesecond set of buffers 620. In some embodiments, the write switch 605 maybe a 2×2 switch configured for double buffering control to receive datafrom two sources and write data to two sets of buffers (e.g., the firstset of buffers 615 and the second set of buffers 620). In otherembodiments, the write switch 605 may be configured in other ways.

The read switch 610 may be configured to read data stored within thefirst set of buffers 615 and the second set of buffers 620. For example,the read switch 610 may read data written by the write switch 605 in thefirst set of buffers 615 and/or the second set of buffers 620 to sendthe read data to the DRAM 215 (via the direct memory access controller245). Similarly, the read switch 610 may read data written by the writeswitch 605 in the first set of buffers 615 and/or the second set ofbuffers 620 to send the read data to the sparse tensor compute cluster(and particularly the sparse tensor compute unit) of the sparse tensorcompute cluster that is associated with the sparse tensor feature mapmemory unit 600. For example, the write switch 605 may receive asub-feature map (and corresponding index values) from the DRAM 215 andstore the sub-feature map (and the index values) within the first set ofbuffers 615 and/or the second set of buffers 620. The read switch 610may then read that sub-feature map (and the index values) from the firstset of buffers 615 and/or the second set of buffers 620 and send theread data to the sparse tensor compute cluster. Similarly, the writeswitch 605 may receive an output sub-feature map from the sparse tensorcompute cluster and write that output sub-feature map within the firstset of buffers 615 and/or the second set of buffers 620. The read switch610 may read that output from the first set of buffers 615 and/or thesecond set of buffers 620 and transmit that output tensor to the DRAM215.

In some embodiments, the read switch 610 may also be 2×2 switchconfigured for double buffering control to read data from two sets ofbuffers (e.g., the first set of buffers 615 and the second set ofbuffers 620). In other embodiments, the read switch 610 may be a 1×1switch configured to read data from a single set of buffers or the readswitch may be configured to read data from more than two sets ofbuffers.

Each of the first set of buffers 615 and the second set of buffers 620may include two buffers in some embodiments. For example, in someembodiments, the first set of buffers 615 may include a first valuebuffer 625 and a first indices buffer 630. Similarly, in someembodiments, the second set of buffers 620 may include a second valuebuffer 635 and a second indices buffer 640. Although only two sets ofbuffers (e.g., the first set of buffers 615 and the second set ofbuffers 620) are shown in the sparse tensor feature map memory unit 600,in other embodiments, a single set of buffers or greater than two setsof buffers may be provided depending upon the configuration of the writeswitch 605 and/or the read switch 610. Similarly, although each of thefirst set of buffers 615 and the second set of buffers 620 is shown tohave two buffers each, in other embodiments, either or both of the firstset of buffers and the second set of buffers may include greater thantwo buffers or possibly a single buffer each.

The first value buffer 625 and the second value buffer 635 may beconfigured to store data values of the sub-feature map or the outputsub-feature map, while the first indices buffer 630 and the secondindices buffer 640 may be configured to store the index values of thesub-feature maps or the output sub-feature maps. For example, in someembodiments, the data values of sub-feature map A may be stored withinthe first value buffer 625 and the index values of that sub-feature mapmay be stored within the first indices buffer 630. In other embodiments,the data values of a particular sub-feature map (or the outputsub-feature map) may be stored within one of the first set of buffers615 or the second set of buffers 620 and the index values of thatparticular sub-feature map (or output sub-feature map) may be storedwithin the other one of the first set of buffers or the second set ofbuffers. Further, in some embodiments, the first set of buffers 615 maybe designated to store the data values and the index values of thesub-feature maps, while the second set of buffers may be configured tostore the data values and index values of the output sub-feature maps.In other embodiments, each of the first set of buffers 615 and thesecond set of buffers 620 may store both—the sub-feature maps and theoutput sub-feature maps (and their corresponding index values). Thus,each sub-feature map and each output sub-feature map may be associatedwith two buffers—a value buffer (e.g., the first value buffer 625, thesecond value buffer 635) to store the data values of the sub-feature mapor the output sub-feature map and an indices buffer (e.g., the firstindices buffer 630, the second indices buffer 640) to store the indexvalues of that sub-feature map or the output sub-feature map.

Additionally, although the data values and the index values of aparticular sub-feature map or the output sub-feature map are shown asbeing stored in separate buffers (e.g., the first value buffer 625, thesecond value buffer 635, the first indices buffer 630, the secondindices buffer 640), in some embodiments, the data values and the indexvalues of a particular sub-feature map or the output sub-feature map maybe stored within a single buffer. In other words, in some embodiments,the first value buffer 625 and the first indices buffer 630 may bemerged together to form a single buffer. Similarly, in some embodiments,the second value buffer 635 and the second indices buffer 640 may bemerged together to form a single buffer.

Each buffer in the first set of buffers 615 and the second set ofbuffers 620 may be an SRAM memory configured as a single port read/writeregister file, a first-in-first-out data structure, a set of registers,or the like. By using SRAM memory for the buffers in the first set ofbuffers 615 and the second set of buffers 620, complex and moreexpensive cache structures may be avoided. In other embodiments, one ormore buffers in the first set of buffers 615 and/or the second set ofbuffers 620 may be other types of memories. Further, each buffer in thefirst set of buffers 615 and the second set of buffers 620 may beconfigured with a particular size to be able to accommodate the datavalues and index values of at least one sub-feature map or at least oneoutput sub-feature map.

Referring now to FIG. 7, an example sparse tensor weight memory 700 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor weight memory 700 is analogous to the sparse tensorweight memory 410 and the sparse tensor weight memory 510. The sparsetensor weight memory 700 includes a first buffer 705 configured to storethe weight values and a second buffer 710 to store the index values ofthe weight values from the weight matrix 220. Thus, the first buffer 705is similar to the first value buffer 625 and the second value buffer635, while the second buffer 710 is similar to the first indices buffer630 and the second indices buffer 640. The first buffer 705 and thesecond buffer 710 may receive and store weight values/index values fromthe DRAM 215 and send those values to the sparse tensor compute cluster(e.g., the sparse tensor compute cluster 230). In some embodiments,greater than one buffer for storing the weight values and/or greaterthan one buffer for storing the index values of the weight matrix may beused.

Turning to FIG. 8, an example sparse tensor compute cluster 800 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor compute cluster 800 is analogous to the sparse tensorcompute cluster 230. The sparse tensor compute cluster 800 includes aplurality of sparse tensor compute units 805A-805M. The number of theplurality of sparse tensor compute units 805A-805M may be dependent upona designated number of sub-feature maps that may be desired to beprocessed in parallel. For example, to process five sub-feature maps inparallel, five sparse tensor compute units may be provided, with eachsparse tensor compute unit being configured to process one sub-featuremap at a time. In some embodiments, the number of the plurality ofsparse tensor compute units 805A-805M is same as the number of theplurality of sparse tensor feature map memory units 415A-415M or515A-515M, with a sparse tensor compute unit #i being associated withsparse tensor feature map memory unit #i. In other embodiments,different numbers of the plurality of sparse tensor compute units805A-805M and the plurality of sparse tensor feature map memory units415A-415M or 515A-515M may be used.

Further, in some embodiments, each of the plurality of sparse tensorcompute units 805A-805M may be independent from other sparse tensorcompute units, and process data independent from other ones of theplurality of sparse tensor compute units. Each of the plurality ofsparse tensor compute units 805A-805M receives a sub-feature map (andcorresponding index values) from the plurality of sparse tensor featuremap memory units 415A-415M or 515A-515M via a bi-directional bus810A-810M, respectively. The bi-directional bus 810A-810M may also beused to send the output sub-feature maps back to the plurality of sparsetensor feature map memory units 415A-415M or 515A-515M.

For example, if the sparse tensor memory cluster 400 having the staticbinding is used, in some embodiments, the sparse tensor compute unit #imay be configured to receive the sub-feature map (and correspondingindex values) stored within the sparse tensor feature map memory unit #ivia the bi-directional bus #i. In such embodiments, the sparse tensorcompute unit #i may also be configured to send the output sub-featuremap to the sparse tensor feature map memory unit #i via thebi-directional bus #i. Thus, a one-to-one correspondence between aparticular sparse tensor compute unit and a sparse tensor feature mapmemory unit exists. For example, during static binding, the sparsetensor compute unit 805A may receive a sub-feature map (andcorresponding index values) from the sparse tensor feature map memoryunit 415A via the bi-directional bus 810A, and send the resulting outputsub-feature map back to the sparse tensor feature map memory unit 415Avia the bi-directional bus 810A.

In other embodiments, if the sparse tensor memory cluster 500 is used, asparse tensor compute unit #i may still be associated with sparse tensorfeature map memory unit #i. However, depending upon the configuration ofthe memory interconnect 525, a sparse tensor compute unit #i may receivea sub-feature map from a sparse tensor feature map memory unit #(i+1)−Mvia the bi-directional bus 810A-810M. Further, depending upon theconfiguration of the memory interconnect 525, a sparse tensor computeunit #i may be able to send the output sub-feature map to a sparsetensor feature map memory unit #(i+1)−M in addition to the sparse tensorfeature map memory unit #i.

In addition to the sub-feature maps, each of the plurality of sparsetensor compute units 805A-805M receives weight values (and correspondingindex values) via a uni-directional bus 815A-815M from the sparse tensorweight memory 410 or the sparse tensor weight memory 510. In someembodiments, the same weight may be transmitted to each, or at least agroup, of the plurality of sparse tensor compute units 805A-805M via theuni-directional bus 815A-815M. In other embodiments, different weightsmay be transmitted to each, or at least a group, of the plurality ofsparse tensor compute units 805A-805M via the uni-directional 815A-815M.Further, in some embodiments, a single weight may be transmitted to theplurality of sparse tensor compute units 805A-805M at a time, while inother embodiments, more than one weight may be simultaneouslytransmitted to one or more of the plurality of sparse tensor computeunits at a time.

The output sub-feature maps obtained from processing a sub-feature mapmay be transmitted back to the corresponding one of the sparse tensorfeature map memory unit. For example, in some embodiments, the sparsetensor compute unit 805A may receive a sub-feature map from the sparsetensor feature map memory unit 415A or 515A, process the sub-feature mapto obtain an output sub-feature map, and send the output sub-feature mapback to the sparse tensor feature map memory unit 415A or 515A. Thesparse tensor feature map memory unit 415A or 515A may then send theoutput sub-feature map to the DRAM 215, to another sparse tensor featuremap memory unit, and/or to another sparse tensor compute unit based uponthe configuration.

Turning to FIG. 9, an example sparse tensor compute cluster 900 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor compute cluster 900 is analogous to the sparse tensorcompute cluster 230. The sparse tensor compute cluster 900 is alsosubstantially similar to the sparse tensor compute cluster 800. Forexample, similar to the sparse tensor compute cluster 800, the sparsetensor compute cluster 900 includes a plurality of sparse tensor computeunits 905A-905M connected via a bi-directional bus 910A-910M to at leastone of the sparse tensor feature map memory units 415A-415M or515A-515M, as discussed above. Also similar to the sparse tensor computecluster 800, each of the plurality of sparse tensor compute units905A-905M is connected via a uni-directional bus 915A-915M to the sparsetensor weight memory 410 or 510 to receive the weights.

However, unlike the sparse tensor compute cluster 800 in which each ofthe plurality of sparse tensor compute units 805A-805M is independentand does not share the data being processed therein with other ones ofthe plurality of sparse tensor compute units, the plurality of sparsetensor compute units 905A-905M of the sparse tensor compute cluster 900are interconnected via an interconnect 920. The interconnect 920 may beconfigured to override the static binding discussed above. Thus, in someembodiments, the interconnect 920 may enable a sparse tensor computeunit #i to communicate with other ones of the sparse tensor computeunits #1−M depending upon the configuration of the interconnect. Forexample, in some embodiments, the interconnect 920 may be two-by-twoswitch that enables a sparse tensor compute unit #i to communicate withthe sparse tensor compute unit #i+1. In other embodiments, theinterconnect 920 may be a multi-stage interconnect such as a meshnetwork or Benes Network that allows a sparse tensor compute unit #i tocommunicate with each of the other sparse tensor compute units #1−M.

Each of the plurality of sparse tensor compute units 905A-905M may beconnected to the interconnect 920 via a bi-directional bus 925A-925M.Thus, each of the plurality of sparse tensor compute units 905A-905M maybe configured to send the output sub-feature map resulting fromprocessing a particular sub-feature map directly to another one of theplurality of sparse tensor compute units via the interconnect 920 andthe bi-directional bus 925A-925M instead of first sending those resultsto the sparse tensor memory cluster 400 or 500. By using theinterconnect 525, the flexibility in assigning and processingsub-feature maps may be increased.

Turning to FIG. 10, an example block diagram of a sparse tensor computeunit 1000 is shown, in accordance with some embodiments of the presentdisclosure. The sparse tensor compute unit 1000 is analogous to one ofthe plurality of sparse tensor compute units 805A-805M or 905A-905M. Thesparse tensor compute unit 1000 is configured to perform various machinelearning operations such as multiplication, addition, etc., that mayneed to be performed during a convolution operation in a CNN. Thus, thesparse tensor compute unit 1000 receives a sub-feature map (andcorresponding index values) from the sparse tensor memory cluster 400 or500, as discussed above, or an output sub-feature map from anothersparse tensor compute unit. The sparse tensor compute unit 1000 alsoreceives weight values from the sparse tensor weight memory 410 or 510.

The sparse tensor compute unit 1000 includes a plurality of multipliers1005A-1005P, each of which is configured to multiply a data value of asub-feature map with a weight value of the weight matrix 220. In someembodiments, the number of the plurality of multipliers 1005A-1005P maybe dependent upon the number of cells in the sub-feature map. Forexample, for a 2×2 sub-feature map having a total of four cells acrosstwo rows and two columns, in some embodiments, the number of theplurality of multipliers 1005A-1005P may be four to enable the datavalue in each cell to be processed independently. In other embodiments,the number of the plurality of multipliers 1005A-1005P may be greaterthan or less than the number of cells in a sub-feature map.

For example, in some embodiments, the number of the plurality ofmultipliers 1005A-1005P may be dependent upon the number of cells havingnon-zero values in the sub-feature map. For example, in the 2×2sub-feature map above having four cells, if only three cells havenon-zero values, the number of the plurality of multipliers 1005A-1005Pmay be three to process the three non-zero values in parallel.Alternatively, the number of the plurality of multipliers 1005A-1005Pmay still be four, however, only three of the four multipliers may beengaged/used to process the three non-zero values. In other embodiments,the number of the plurality of multipliers 1005A-1005P may be two, andtwo of the three non-zero values may be processed in parallel in a firstround and the third non-zero value may be processed in a second round inone of the two multipliers after the first round.

In some embodiments, all of the P multiplier units may be used in eachclock cycle. For example, if the number of the plurality of multipliers1005A-1005P is 4 and there are 3 non-zero values (d1,d2,d3) in a 2×2cell on which weights w1 and w2 are to applied. At clock cycle 1, all 4multipliers may be utilized as follows: d1*w1, d2*w1, d3*w1 and d1*w2.Generally speaking, if P is the total number of data values in thesub-feature map being processed by the sparse tensor compute cluster1000 and Q is the number of non-zero values in the sub-feature map, insome embodiments (e.g., in CONV2 layers), Q multipliers may be engagedand max(1, ceiling(P−Q)/Q) unique weights may be transmitted to the Qmultipliers to ensure full utilization of the plurality of multipliers1005A-1005P. For example, in case of 1 unique weight value, the uniqueweight value may be transmitted to each of the Q multipliers. In thecase of 2 unique weight values, the first weight value may betransmitted to the Q multipliers and the second weight value may betransmitted to the remaining number of multipliers. In the case of 1×1CONV and FC layers, P weight values may be transmitted to P multipliers.Thus, in some embodiments, only non-zero data values of the sub-featuremap are input into the plurality of multipliers 1005A-1005P. Since aproduct with a zero data value is zero, any zero values in the inputsub-feature map need not be processed through the plurality ofmultipliers 1005A-1005P, thereby saving computing resources and time.Rather, in some embodiments, after the computation results of thenon-zero values are filled in the output feature map, the remainingindex values may be filled with zero values.

In some embodiments, the number of the plurality of multipliers1005A-1005P may be dependent upon the size of the weight matrix 220(also referred to herein as a kernel matrix or filter matrix). Theweight matrix 220 may also include a plurality of cells, as discussedabove. For example, the weight matrix 220 may have two rows and twocolumns forming four cells. Thus, the number of the plurality ofmultipliers 1005A-1005P that are provided or engaged may be four toprocess four data values in parallel. Thus, the number of the pluralityof multipliers 1005A-1005P may be dependent upon a variety of factors.

In some embodiments, only non-zero weight values may be input into theplurality of multipliers 1005A-1005P. Thus, for example, if the weightmatrix 220 has four cells and only three of those cells have non-zerovalues, only three weight values may be input into the plurality ofmultipliers 1005A-1005P. In some embodiments, the number of non-zeroweight values may not impact the number of the plurality of multipliers1005A-1005P, but rather, may impact the number of iterations that areneeded to process a sub-feature map, as discussed below. Similar to thezero data values, the appropriate index values of where the result ofmultiplying with that zero-weight value would be located in the outputsub-feature map may be computed. Those index values may then bepopulated with a value of zero.

Further, in some embodiments, each sparse tensor compute unit (e.g., theplurality of sparse tensor compute units 805A-805M, 905A-905M) in asparse tensor computer cluster (e.g., the sparse tensor compute cluster800, 900) may have the same number of multipliers (e.g., the pluralityof multipliers 1005A-1005P), while in other embodiments, the number ofmultipliers in one or more sparse tensor compute units of a sparsetensor compute cluster may be different than other ones of the sparsetensor compute units in the sparse tensor compute cluster.

Additionally, each of the plurality of multipliers 1005A-1005P may besized based upon the size of the data values and the weight values thatare to be processed therein. In some embodiments, each of the pluralityof multipliers 1005A-1005P may be an electronic circuit configured tomultiply two binary numbers. Generally speaking, each of the pluralityof multipliers 1005A-1005P may be implemented in any of a variety ofways using software, hardware, firmware, or combinations thereof.

Each of the plurality of multipliers 1005A-1005P, thus, receives a datavalue 1010A-1010P from a sub-feature map. For example, and referring toFIG. 11 in conjunction with FIG. 10, the sparse tensor compute unit 1000is explained with respect to an example 5×5 input feature map 1100. Itis to be understood that the input feature map 1100 is simply an exampleand is not intended to be limiting in any way. The present disclosuremay be used to process input feature maps of any size and having anydata values therein. The input feature map 1100 may be generated fromthe input image 205. The input feature map 1100 may include a pluralityof cells, each cell being formed at the intersection of a row (thatextends in an X-direction 1105) and a column (that extends in aY-direction 1110). Each of the plurality of cells in the input featuremap 1100 includes a data value that is to be processed in the sparsetensor compute unit 1000.

In some embodiments, the input feature map 1100 may be padded by zeroeson all sides to ensure that the output feature map is of the same sizeas the input feature map. For example, a row of zeroes may be addedabove the first row and below the last row of the input feature map1100, and a column of zeroes may be added to the left hand side of thefirst column and to the right hand side of the last column of that inputfeature map for padding. By padding zeroes to the input feature map1100, a padded feature map 1115 having a 7×7 size may be obtained. Insome embodiments, the partitioning block 210 may perform the padding. Inother embodiments, the padding may be added by another component of theaccelerator 200. Without zero padding, the output feature map may be ofa different size than the input feature map.

Further, in some embodiments, sub-feature maps may be created from thepadded feature map 1115, and weight values from a kernel matrix 1120 maybe applied to those sub-feature maps. It is to be understood that thekernel matrix 1120 is simply an example and is not intended to belimiting in any way. The kernel matrix 1120 may assume other sizes(e.g., the number of rows and number of columns may vary from thatshown) and the values within the kernel matrix may also vary. The kernelmatrix 1120 may be said to be of a kernel size, k. In a square kernelmatrix (e.g., the kernel matrix 1120) having equal number of rows andcolumns, the kernel size, k, is equal to the number of rows or columnsin the kernel matrix in some embodiments. In other embodiments, thekernel size, k, for a square or non-square kernel matrix may beconsidered an input parameter that may be determined/optimized usingexternal training processes involving supervised examples andbackpropagation of error gradients. Thus, for the kernel matrix 1120having two rows and two columns, the description below is assuming thatthe kernel size, k, is two.

In some embodiments, the padded feature map 1115 may be divided intosub-feature maps in the partitioning block 210 or other component of theaccelerator 200, as discussed above in FIGS. 3A and 3B. For example andas shown in FIG. 11, the padded feature map 1115 may be divided, asdiscussed above, to create a plurality of sub-feature maps 1125A-1125I.Since the last row and the last column of the padded feature map 1115includes all zero values, those values do not have an impact on theoutput values, and therefore, need not be processed. Each of theplurality of sub-feature maps 1125A-1125I may be processed in parallelin different sparse tensor compute units (e.g., the sparse tensorcompute unit 1000). For example, in some embodiments, the sub-featuremap 1125A may be processed in a first sparse tensor compute unit, thesub-feature map 1125B may be processed in a second sparse tensor computeunit, and so on.

Further, in FIG. 11, each of the plurality of sub-feature maps1125A-1125I includes two rows and two columns. Although each of theplurality of sub-feature maps 1125A-1125I is of the same size as othersub-feature maps in FIG. 11, it is to be understood that the sub-featuremaps may be of varying sizes, as discussed above. Further, although eachof the plurality of sub-feature maps 1125A-1125I has the same number ofrows and the columns as the kernel matrix 1120, in some embodiments, oneor more of the plurality of sub-feature maps may have varying number ofrows and/or columns than the kernel matrix. Depending upon the number ofthe plurality of multipliers 1005A-1005P in the sparse tensor computeunit 1000, multiple data values in each of the plurality of sub-featuremaps 1125A-1125I may be processed in parallel. For example, if thesub-feature map 1125A is being processed in the sparse tensor computeunit 1000, and assuming that the plurality of multipliers 1005A-1005Pincludes four multipliers at least, the data values in each of the fourcells of that sub-feature map may be processed in parallel.

Thus, to process the sub-feature map 1125A, the data values from thatsub-feature map may be input into the plurality of multipliers1005A-1005P. For example, the data value “0” having the index value(1,1) (e.g., row 1, column 1) of the sub-feature map 1125A may be loadedinto the multiplier 1005A, the data value “0” having the index value(1,2) (e.g., row 1, column 2) may be loaded into the multiplier 1005B,the data value “1” having the index value (2,1) (e.g., row 2, column 1)may be loaded into the multiplier 1005C, and the data value “5” havingthe index value (2,2) (e.g., row 2, column 2) may be loaded into themultiplier 1005P. In some embodiments, the corresponding index values ofthe data values may also be input into the respective one of theplurality of multipliers 1005A-1005P. The data values that are zero invalue are loaded into the plurality of multipliers 1005A-1005P hereinsimply for ease of illustration. In other embodiments, only non-zerodata values may be loaded into the plurality of multipliers 1005A-1005P.

In addition to the data values, each of the plurality of multipliers1005A-1005P receives a weight value 1015A-1015P from the sparse tensorweight memory 410 or 510. The weight value 1015A-1015P may be a valuefrom a cell of the kernel matrix 1120. In some embodiments, the sparsetensor compute unit 1000 may be configured to process one unique weightat a time. In such cases, a single weight value may be broadcast to eachof the plurality of multipliers 1005A-1005P at a time. For example, in afirst iteration of processing the sub-feature map 1125A, a first weightvalue from the kernel matrix 1120 may be transmitted to each of themultipliers 1005A, 1005B, 1005C, and 1005P (collectively referred tobelow as the plurality of multipliers 1005A-1005P). Thus, the sameweight value is broadcast to each of the plurality of multipliers1005A-1005P.

Upon finishing processing of the sub-feature map 1125A with the firstweight value, a second weight value from the kernel matrix 1120 may betransmitted to each of the plurality of multipliers 1005A-1005P in asecond iteration. Upon finishing processing of the sub-feature map 1125Awith the second weight value, a third weight value from the kernelmatrix 1120 may be transmitted to each of the plurality of multipliers1005A-1005P in a third iteration, and upon finishing processing of thesub-feature map with the third weight value, a fourth weight value fromthe kernel matrix may be transmitted to each of the plurality ofmultipliers in a fourth iteration. Thus, the processing of thesub-feature map 1125A may require four iterations. During eachiteration, the input weight value is multiplied with each of the datavalues in the sub-feature map 1125A. Further, each iteration may includeone or more rounds depending upon the number of the plurality ofmultipliers 1005A-1005P. Specifically, if the number of the plurality ofmultipliers 1005A-1005P includes enough number of multipliers to processall the data values of a sub-feature map in parallel, then eachiteration may include a single round. On the other hand, if the numberof the plurality of multipliers 1005A-1005P is less than the number ofdata values in the sub-feature map, then each iteration may includemultiple rounds. Upon completing the four iterations with thesub-feature map 1125A, the output of the sparse tensor compute unit 1000may be the output sub-feature map corresponding to the sub-feature map1125A.

Further, in some embodiments, the order in which the weight values fromthe kernel matrix 1120 are transmitted to the plurality of multipliers1005A-1005P may be pre-determined. For example, in some embodiments, theweight value from the kernel matrix 1120 having index value (1,1) may beloaded into each of the plurality of multipliers 1005A-1005P in thefirst iteration. As indicated above, the index value captures the rownumber and column number of a particular cell in the kernel matrix 1120.Thus, the index value (1,1) corresponds to row 1 column 1 of the kernelmatrix 1120. The weight value corresponding to the index value (1,1) inthe kernel matrix 1120 is “0.” Thus, the weight value of “0” is loadedinto each of the plurality of multipliers 1005A-1005P in the firstiteration. Similar to the data values, zero weight values are describedas being input into the plurality of multipliers 1005A-1005P simply forease of illustration. In other embodiments, only non-zero weight valuesmay be input into the plurality of multipliers 1005A-1005P. In thesecond iteration, the weight value in the index value (1,2) (e.g., theweight value “1”) is broadcast to each of the plurality of multipliers1005A-1005P. In the third iteration, the weight value corresponding tothe index value (2,2) (e.g., having the weight value “0”) is loaded intoeach of the plurality of multipliers 1005A-1005P, while in the fourthiteration, the weight value of “−1” corresponding to the index value(1,1) is broadcast to each of the plurality of multipliers. In otherembodiments, the weight values may be input into the plurality ofmultipliers 1005A-1005P in a different order in the various iterations.

Thus, in the first iteration of processing the sub-feature map 1125A,each of the plurality of multipliers 1005A-1005P receives one data valuefrom the sub-feature map and the weight value of “0” corresponding tothe index value (1,1) in the kernel matrix 1120. Each of the pluralityof multipliers 1005A-1005P computes a product (e.g., z=w*x) between itsrespective data value (e.g., the data values 1010A-1010P) and the weightvalue (e.g., the weight values 1015A-1015P) to obtain a product. Forexample, the multiplier 1005A may multiply the data value 1010A with theweight value 1015A to obtain a product value 1020A. Similarly, themultiplier 1005B may multiply the data value 1010B with the weight value1015B to obtain a product value 1020B, the multiplier 1005C may multiplythe data value 1010C with the weight value 1015C to obtain a productvalue 1020C, and the multiplier 1005P may multiply the data value 1010Pwith the weight value 1015P to obtain a product value 1020P. The productvalues 1020A-1020P may be represented as a product matrix such that eachof the product values 1020A-1020P has the same index value as the datavalue that was input into the respective one of the plurality ofmultipliers 1005A-1005P. For example, since the data value having theindex value of (1,1) was input into the multiplier 1005A, the productvalue 1020A also has the index value of (1,1) in the product matrix.Thus, the product matrix based on the product values 1020A-1020P maylook like:

$\quad\begin{matrix}0 & 0 \\0 & 0\end{matrix}$

Each of the product values 1020A-1202P is input into a correspondingaccumulator 1025A-1025P, respectively. Each of the accumulators1025A-1025P may include a register (or other type of memory) configuredto receive and temporarily store the respective one of the productvalues 1020A-1020P. In some embodiments, each of the accumulators1025A-1025P may also include a computing element (e.g., an additionelement) to perform the computations discussed below. Although Pseparate ones of the accumulators 1025A-1025P are shown in FIG. 10, insome embodiments, one or more of the P accumulators may be combinedtogether to form a larger accumulator that receives the product values1020A-1020P. Additionally, in some embodiments, the size of each of theaccumulators 1025A-1025P may be computed using the formula: (2k−1) x(2k−1), where k is the kernel size of the kernel matrix 1120. Since inthe current example, the kernel size, k, of the kernel matrix 1120 is 2,each of the accumulators 1025A-1025P may be sized to at least store a3×3 matrix.

Further, upon transmitting the product values 1020A-1020P to theaccumulators 1025A-1025P, the second iteration of processing thesub-feature map 1125A may start in the plurality of multipliers1005A-1005P. Thus, in the second iteration, the weight valuecorresponding to the index value (1,2) of the kernel matrix 1120 may betransmitted to each of the plurality of multipliers 1005A-1005P. Sincethe data values of the sub-feature map 1125A are already in theplurality of multipliers 1005A-1005P from the first iteration, thosedata values need not be input again.

Further, upon receiving the product values 1020A-1020P, each of theaccumulators 1025A-1025P may process those values based upon valuesreceived from a register bank 1030. The register bank 1030 is configuredto store the output sub-feature map resulting from convolving thesub-feature map 1125A with the kernel matrix 1120. The register bank1030 may be a group of registers, flip flops, or other memory units.While the register bank 1030 is explained here in terms of registers, inother embodiments, flip flops or other types of memory units may be usedin the register bank. In some embodiments, the register bank 1030 may bepart of one or more of the accumulators 1125A-1125P. Further, in someembodiments, the register bank 1030 may include a plurality of rowregisters and a plurality of column registers that are connectedtogether to form one or more shift registers. In some embodiments, theplurality of row registers may be connected together to form a shiftregister to enable the values stored therein to be shifted right or leftby at least one position. Similarly, in some embodiments, the pluralityof column registers may be connected together to form a shift registerto enable the values stored therein to be shifted up or down by at leastone position.

Further, the size of the register bank 1030 may be based upon the sizeof the padded feature map 1115 (or the size of the output feature map).Thus, for a 7×7 size of the padded feature map 1115, the register bank1030 may be initialized with a size of 7×7. In other words, the registerbank 1030 may include 49 registers such that 7 registers in each row areconnected together to form a shift register and 7 registers in eachcolumn are connected together to form a shift register. Further, in someembodiments, at the start of the first iteration, the register bank 1030may be initialized with all zero values, as shown in register bank 1130in FIG. 11. Additionally, in some embodiments, at the start of the firstiteration, a portion of the register bank 1130 may be loaded into eachof the accumulators 1025A-1025P.

For example, each of the accumulators 1025A-1025P may include a register1135 having a size of (2k−1)×(2k−1) or 3×3, as discussed above. When theregister bank 1130 is initialized with all 0 values at the beginning ofthe first iteration, the register 1135 of each of the accumulators1025A-1025P may be loaded with values from a 3×3 portion of the registerbank 1130. The 3×3 portion from the register bank 1130 that is copiedinto the register 1135 may be determined based on the following formula:

Register=register bank(i:i+acc_length−1,j:j+acc_length−1)

In the formula above, the first term corresponds to the row numbers ofthe register bank 1130 and the second term corresponds to the columnnumbers of the register bank. Further, in the formula above, “i” is thestart of the row number and “j” is the start of the column number of thesub-feature map 1125A in the padded feature map 1115, and acc_length isthe size of the register 1135. In other words, (i, j) is the index valueof the sub-feature map 1125A. For example, the size of the register 1135in the example above is 3 since the register is 3×3. The size of theregister 1135 is computed similar to the kernel size of the kernelmatrix 1120. Thus, for the sub-feature map 1125A, “i” in the formulaabove is 1, “j” is 1, and acc_length is 3. Thus, the register 1135 ofeach of the accumulators 1025A-1025P may be initialized with rows (1:3,1:3) of the register bank 1130. Since the values in rows 1:3 and columns1:3 of the register bank 1130 are all zeroes at the time ofinitialization, the values in the register 1135 are initialized with allzeroes. In addition to the product values 1020A-1020P, the register bank1030 also receives the index values corresponding to each data value inthe sub-feature map 1125A and the index values corresponding to eachweight value in the kernel matrix 1120 to compute the portion of theregister bank 1030 that is to be loaded into each of the accumulators1025A-1025P.

Thus, at the start of the first iteration, the register bank 1030 isinitialized with the 0 values shown in the register bank 1130 and theregister 1135 of each of the accumulators 1025A-1025P is initializedwith a 3×3 portion from the register bank 1130. Each of the accumulators1025A-1025P reads a portion of its respective register 1135 in eachiteration and adds the current product values (e.g., the product values1020A-1020P) to the product values computed in previous iterations. Forexample, for the first iteration in which the weight value correspondingto the index value (1,1) is transmitted to the plurality of multipliers1005A-1005P, the accumulators 1025A-1025P read a portion 1140A of theirrespective instance of the register 1135, as shown in FIG. 11. The orderin which portions of the register 1135 are read may be determined by thedefinition of the convolution operation. Thus, in some embodiments, theportion 1140A is read first. In other embodiments, depending upon thedefinition of the convolution operation, another portion of the register1135 may be read first. The accumulators 1025A-1025P add the productvalues 1020A-1020P in the product matrix above with the values in theportion 1140A. Specifically, the accumulators 1025A-1025P may performthe following matrix addition:

${\begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix} + \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}} = \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}$

In the equation above, the first matrix is the product matrix derivedfrom the product values 1020A-1020P and the second matrix corresponds tothe values in the portion 1140A. The result of the matrix addition maybe stored back into the portion 1140A of the register 1135 of each ofthe accumulators 1025A-1025P. Thus, after the first iteration, theregister 1135 of each of the accumulators 1025A-1025P has the valuesshown in register 1145A of FIG. 11.

In the second iteration, the sub-feature map 1125A continues to beloaded into the plurality of multipliers 1005A-1005P, as discussedabove, and the weight value of “1” corresponding to the index (1, 2) inthe kernel matrix 1120 is transmitted to each of those multipliers. Theproduct values 1020A-1020P may be represented by combining the productvalues 1020A-1020P in a product matrix, z in the second iteration as:

$\quad\begin{matrix}0 & 0 \\0 & 5\end{matrix}$

The above product values 1020A-1020P may be transmitted to theaccumulators 1025A-1025P. The accumulators 1025A-1025P may read aportion 1140B from the register 1145A, which is obtained by shiftingfrom the location of the portion 1140A left column-wise by one position.The accumulators 1025A-1025P may perform matrix addition on the valuesin the product matrix of the second iteration with the values in theportion 1140B as follows:

${\begin{bmatrix}0 & 0 \\0 & 5\end{bmatrix} + \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}} = \begin{bmatrix}0 & 0 \\0 & 5\end{bmatrix}$

The result of the matrix addition above may be stored back in theportion 1140B. Thus, after the second iteration, the register 1135 hasvalues shown in register 1145B:

In the third iteration, the sub-feature map 1125A continues to be loadedinto the plurality of multipliers 1005A-1005P, as discussed above, andthe weight value of “1” corresponding to the index (2, 2) in the kernelmatrix 1120 is transmitted to each of those multipliers. The productvalues 1020A-1020P may be represented by a product matrix, z in thesecond iteration as:

$\quad\begin{matrix}0 & 0 \\0 & 0\end{matrix}$

The above product values 1020A-1020P may be transmitted to theaccumulators 1025A-1025P. The accumulators 1025A-1025P may read aportion 1140C of the register 1145B, which is obtained by shifting thelocation of the portion 1140B of the register 1145A up row-wise by oneposition. The accumulators 1025A-1025P may perform matrix addition onthe values in the product matrix of the third iteration with the valuesin the portion 1140C as follows:

${\begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix} + \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}} = \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}$

The result of the matrix addition above may be stored in the portion1140C. Thus, after the third iteration, the register 1135 has the valuesshown in register 1145C.

In the fourth iteration, the sub-feature map 1125A continues to beloaded into the plurality of multipliers 1005A-1005P, as discussedabove, and the weight value of “−1” corresponding to the index (2, 1) inthe kernel matrix 1120 is transmitted to each of those multipliers. Theproduct values 1020A-1020P may be represented by a product matrix, z inthe second iteration as:

$\quad\begin{matrix}0 & 0 \\0 & {- 5}\end{matrix}$

The above product values 1020A-1020P may be transmitted to theaccumulators 1025A-1025P. The accumulators 1025A-1025P may read aportion 1140D of the register 1145C, which is obtained by shifting thelocation of the portion 1140C to the right column-wise by one position.The accumulators 1025A-1025P may perform matrix addition on the valuesin the product matrix of the fourth iteration with the values in theportion 1140D as follows:

${\begin{bmatrix}0 & 0 \\0 & {- 5}\end{bmatrix} + \begin{bmatrix}0 & 0 \\0 & 0\end{bmatrix}} = \begin{bmatrix}0 & 0 \\0 & {- 5}\end{bmatrix}$

The result of the matrix addition above may be stored in the portion1140D. Thus, after the fourth iteration, the register 1135 has thevalues shown in register 1145D.

Since in the example of FIG. 11, there are only four iterations, uponcompleting the four iterations for the sub-feature map 1125A, theaccumulators 1025A-1025P load the values stored in the register 1135(e.g., the values in the register 1145D) back into the register bank1130. Further, the values of the register 1145D may be loaded back intothe register 1130 in the same location from where those values wereinitially initialized. Thus for example, the following formula may beused to determine where the values from the register 1145A are touploaded to the register 1130:

Register bank=register(i:i+acc_length−1,j:j+acc_length−1)

Thus, for the sub-feature map 1125A, the values in the register 1145Dmay be loaded into rows (1:3, 1:3) of the register bank 1130. Thus,after loading the values in the register 1145D to the register bank1130, the register bank 1030 may have values similar to register bank1150. Further, the loaded values in the register bank 1150 may becircular shifted right by one column. For example, the shiftingoperation may be performed in accordance with the following formula:

Register bank=circshift(register bank,kernel size−1,2)

The directionality of shifting may be dependent upon the convolutiondefinition. Thus, in other embodiments, the register bank 1150 may beshifted in other ways. Thus, the values in the register 1150 may beshifted to obtain values shown in register 1155. The values in theregister 1150 may be subject to one or more in-line operations in theregister bank 1030. For example, the register bank 1030 may perform anon-linear Rectified Linear Unit (ReLU) operation and a poolingoperation. Thus, in some embodiments, the register bank 1030 may includea ReLU processing unit and pooling operation processing unit to performthe ReLU and pooling operations, respectively. In other embodiments, aseparate component may be associated with the sparse tensor compute unit1000 to perform the ReLU and pooling operations.

In some embodiments, a ReLU operation may implement the followingactivation function: ƒ(x)=max (0,x). In other embodiments, the ReLUoperation may implement other activation functions. The ReLU operationmay generate one output for each input. Thus, for A inputs, the ReLUoperation may generate A outputs. A pooling operation may then reducethe A results to B results. For example, a pooling operation having akernel size of 2×2 may reduce A inputs into A/4 inputs. Thus, dependingupon the kernel size of the pooling operation, the register bank 1030may reduce A inputs into B inputs. The register bank 1130 may also beassociated with B pooling units, with each pooling unit configured toperform a down sampling operation on one of the A results of the ReLUoperation. The pooling operation may perform a max pooling operation inwhich a maximum value from a group of cell values is determined, anaverage pooling operation in which an average of a group of cell valuesis determined, or a sum pooling operation in which a sum of a group ofcell values is determined. In other embodiments, other poolingoperations may be performed. The results of the pooling operation mayrepresent the output sub-feature map, which may be sent back to theassociated sparse tensor memory cluster 400, 500 or to another sparsetensor compute unit. In some embodiments, the output sub-feature map maybe compressed before being sent to the sparse tensor memory cluster 400,500.

In some embodiments, before compressing the output sub-feature map, theoutput sub-feature map may be combined with other output sub-featuremaps generated from the other sparse tensor compute units. To combinethe various output sub-feature maps, in some embodiments, the outputsub-feature maps may be “stitched” together to obtain the output featuremap. For example, in some embodiments, if the input feature map isdivided into four sub-feature maps [A, B, C, D], which generate fourrespective output sub-feature maps A′, B′, C′, and D′, the outputfeature maps may be given as [A′, B′, C′, D′].

Further, each of the product values 1020A-1020P is also input into anadder 1035 and stored in a special accumulator 1040. The adder 1035 maybe used to compute sums of groups of the product values 1020A-1020P. Forexample, the adder 1035 may be used to compute sums of groups of P′inputs and P inputs where P′ is less than P. The special accumulator1040 may handle the accumulations that may be needed for fully connectedlayers and 1×1 convolutions, by taking outputs of the adder 1035 andpreviously accumulated results from the register bank 1030.

Turning now to FIG. 12, an example flowchart outlining operations of aprocess 1200 is shown, in accordance with some embodiments of thepresent disclosure. The process 1200 may be implemented in the sparsetensor compute unit 1000. In some embodiments, a controller may beassociated with the sparse tensor compute unit 1000 to perform theprocess 1200. The controller may be associated with a memory for storingcomputer-readable instructions, which may be performed by a processorassociated with the controller. In other embodiments, the schedulingengine 225 may be configured to perform the process 1200. The process1200 may include other or additional operations depending upon theparticular embodiment.

Upon starting at operation 1205, the register bank 1030 is initializedat operation 1210. As indicated above, at the start of the firstiteration, the register bank 1030 may be initialized with all zerovalues. Further, upon initializing the register bank 1030, each of theaccumulators 1025A-1025P may be initialized at operation 1215 such thata portion of the register bank is loaded into each of thoseaccumulators. The portion of the register bank 1030 that is loaded intothe each of the accumulators is based upon the index values of thesub-feature map in the input feature map being processed. At operation1220, data values from the sub-feature map (e.g., the sub-feature map1125A) are input into the plurality of multipliers 1005A-1005P asdiscussed above. In some embodiments, the operations 1210/1215 and 1220may be occur in parallel. At operation 1225, one weight value from thekernel matrix 1220 is transmitted to each of the plurality ofmultipliers 1005A-1005P.

Each of the plurality of multipliers 1005A-1005P computes a productbetween the data value of the sub-feature map received therein and theweight value at operation 1230 to obtain a product value (e.g., theproduct values 1020A-1020P). The product values may be represented as aproduct matrix. At operation 1235, the values in the product matrix isadded to values read from a window of the portion loaded into theaccumulators 1025A-1025P at the operation 1215. For example, in thefirst iteration, the values in the product matrix may be added to theportion 1140A. In the second iteration, the values in the product matrixmay be added to the portion 1140B, while in the third iteration, thevalues in the product matrix may be added to the portion 1140C, and inthe fourth iteration, the values in the product matrix may be added tothe portion 1140D. Thus, in each iteration, the window (e.g., theportions 1140A-1140D) may be shifted by one position (e.g., from theinitial position of the portion 1140A—shift left column wise to theposition of the portion 1140B—shift up row wise to the position of theportion 1140C—shift right column wise to the position of the portion1140D). The shifting pattern described above may be applied even if theregister 1135 is greater than 3×3 in size.

The result of the addition in each iteration is stored back in theaccumulators 1025A-1025P at operation 1240. Then, at operation 1245, itis determined whether all the iterations have been completed. Again, thenumber of iterations may be dependent upon the number of non-zero weightvalues in the weight matrix 1120 and the number of unique weight valuesthat are to be transmitted to the plurality of multipliers 1005A-1005Pin each iteration. For example, for a 2×2 weight matrix, if a singleweight value is transmitted in each iteration, and if all four weightvalues in that weight matrix are non-zero values, then the process 1200may include 4 iterations. If additional iterations remain, the process1200 loops back to the operation 1210 to continue processing the nextiteration. On the other hand, if at the operation 1245, it is determinedthat all the iterations have completed, at operation 1250, theaccumulators 1025A-1025P load the portion obtained at the operation 1240after completion of the last iteration back into the register bank 1030.At operation 1255, the register bank 1030 performs a shift operation,and at operation 1260, a ReLU and pooling operation is performed on theshifted values in the register bank to obtain an output sub-feature map.Optionally, at operation 1265, the output sub-feature map may becompressed. The process 1200 then ends at operation 1270 by sending theoutput sub-feature map to the associated sparse tensor memory cluster400, 500, or to another sparse tensor compute unit.

In some embodiments, where depth separable convolutions are implemented(e.g., where the feature map has multiple channels), the result fromeach channel may be stored as an intermediate output sub-feature map.For example, the sub-feature maps of each channel may perform operations1205-1255, and the result of those operations may be stored as anintermediate output sub-feature map. In some embodiments, ReLU andpooling operations may not be performed on the intermediate outputsub-feature maps. Thus, depending upon the number of channels, multipleintermediate output sub-feature maps may be obtained. For example, forthree channels, three intermediate output sub-feature maps may beobtained. (e.g., depthwise convolution or spatial convolution performedindependently over each channel of an input). Each of the intermediateoutput sub-feature maps may then be combined by applying a 1×1 filteraccording to given hyperparameters of the neural network. (e.g.,Pointwise convolution such as a 1×1 convolution, projecting the channelsoutput by the depthwise convolution onto a new channel space.) The ReLUand pooling operations of the operation 1260 may then be performed onthe combined intermediate output sub-feature maps to obtain the outputsub-feature map.

Referring now to FIG. 13, an example block diagram of a sparse tensorcompute unit 1300 is shown, in accordance with some embodiments of thepresent disclosure. The sparse tensor compute unit 1300 is analogous toone of the plurality of sparse tensor compute units 805A-805M or905A-905M. The sparse tensor compute unit 1300 is configured to performvarious machine learning operations such as multiplication, addition,etc., that may need to be performed during a convolution operation in aCNN. The sparse tensor compute unit 1300, like the sparse tensor computeunit 1000, includes a plurality of multipliers 1305A-1305P. Each of theplurality of multipliers 1305A-1305P is configured similar to theplurality of multipliers 1005A-1005P, and therefore, is not describedagain.

Further, each of the plurality of multipliers 1305A-1305P is configuredto receive a data value (e.g., data values 1310A-1310P) from asub-feature map (e.g., the sub-feature map 1125A), as well as a weightvalue (e.g., weight values 1315A-1315P) from a kernel matrix (e.g., thekernel matrix 1120) that is to be applied to the sub-feature map. Eachof the plurality of multipliers 1305A-1305P computes a product betweenthe data value (e.g., the data values 1310A-1310P) and the weight value(e.g., the weight values 1315A-1315P) stored therein to generate productvalues 1320A-1320P, respectively. Also, similar to the sparse tensorcompute unit 1000, the sparse tensor compute unit 1300 also processesthe sub-feature map 1125A in multiple iterations (e.g., four iterations,as discussed above). In each iteration, a different weight value fromthe kernel matrix 1120 may be applied to the sub-feature map 1125A.

In some embodiments, each of the plurality of multipliers 1305A-1305Pmay also receive the index values of the data values 1310A-1310P,respectively, and the index values of each of the weight values1315A-1315P. For example, the multiplier 1305A may receive the indexvalue of the data value 1310A and the index values of each of the weightvalues from the kernel matrix 1120. Based upon the index values, each ofthe plurality of multipliers 1305A-1305P may determine the index valueof where the product value (e.g., the product values 1320A-1320P)computed by that multiplier is to be located in the output sub-featuremap. In some embodiments, each of the plurality of multipliers1305A-1305P may compute the index value of the product values1320A-1320P using the following formulae:

XW_row_idx=X_row_idx+((W_ROW_LEN−1)−W_row_idx)

XW_col_idx=X_col_idx+((W_COL_LEN−1)−W_col_idx)

In the formulae above, indices start from 0 and XW_row_idx, XW_col_idxare the row, column index values, respectively, of the product values(e.g., the product values 1320A-1320P) in the output sub-feature map.X_row_idx and X_col_idx are the row, column index values, respectively,of the first multiplicand (e.g., the data values 1310A-1310P).W_row_idx, W_col_idx are the row, column index values, respectively, ofthe second multiplicand (e.g., the weight values 1315A-1315P).W_ROW_LEN, W_COL_LEN are the dimensions (e.g., kernel size) of thekernel matrix 1120. Further, W−row_idx may be the same as X_row_idxoffset by the filter row length and filter coefficient row index filtercoefficient row index takes the value from 0 to W_ROW_LEN−1.

Upon computing the index values of the product values 1320A-1320P, eachof the plurality of multipliers 1305A-1305P may transmit theirrespective product value and the computed index value to an accumulator1325. Although the plurality of multipliers 1305A-1305P have beendescribed as computing the index values of the product values1320A-1320P, in some embodiments, the accumulator 1325 may insteadreceive the various index values of the first and second multiplicandsto compute the index values of those product values. In otherembodiments, another component of the accelerator 200 may compute theindex values for the product values 1320A-1320P in the outputsub-feature map.

Thus, each of the product values 1320A-1320P and the computed indexvalues are input into the accumulator 1325. In some embodiments, theaccumulator 1325 may be a combination of “P” accumulators (where “P” isthe total number of the plurality of multipliers 1305A-1305P) and eachaccumulator may be similar to the plurality of accumulators 1025A-1025P.The accumulator 1325 is, thus, configured similar to the accumulators1025A-1025P. Similar to the accumulators 1025A-1025P, the accumulator1325 may be configured to add each of the product values 1320A-1320P tothe sum computed in the last iteration. For example, the accumulator1325 may add the product value 1320A of the current iteration to theproduct value 1320A from the previous iterations. Thus, the accumulator1325 may perform the following operations:

a 1 = a 1 + z 1 a 2 = a 2 + z 2 … aP = aP + zP

In the equations above, z1 is the product value 1320A, z2 is the productvalue 1320B, zP is the product value 1320P, and so on. At the start ofthe first iteration, each of the values a1, a2, . . . , aP may beinitialized to zero. Results 1330A-1330P, including the summationresults (e.g., a1, a2, . . . aP) and the computed index values of eachof those results may be sent from the accumulator 1325 to a Multi-stageInterconnection Network (“MIN”) 1335. The result 1330A may include thesummation result, a1, and the index value computed by the multiplier1305A for the product value 1320A. Similarly, the result 1330B mayinclude the summation result, a2, as well as the index value computed bythe multiplier 1305B for the product value 1320B, and so on.

The MIN 1335 may be used for local interleaving and connecting theaccumulator 1325 to a variable accumulator register (“VAR”) 1340.Additional details of the MIN 1335 may be found in U.S. application Ser.No. 15/627,042, filed on Jun. 19, 2017, the entirety of which isincorporated by reference herein. The MIN 1335 may include a pluralityof control states. In some embodiments, the total number of locationsinput to a READ MIN (e.g., MIN 1355 described below) are(2k−1){circumflex over ( )}2. Since P values need to be accessed out ofthese (2k−1){circumflex over ( )}2 values, the READ MIN (e.g., the MIN1355) may be used. The total number of locations the WRITE MIN (e.g.,the MIN 1335) is connected to are (2k−1){circumflex over ( )}2. Since Pvalues need to be written out of these (2k−1){circumflex over ( )}2values, the WRITE MIN (e.g., the MIN 1335) may be used. In someembodiments, if two of the product values 1320A-1320P have the samecomputed index values, the MIN 1335 may sort the results 1330A-1330P.The WRITE MIN (e.g., the MIN 1335) may, thus, be used to write to theVAR 1340 and the READ Min (e.g., the MIN 1355) may be used to read fromthe VAR 1340. Thus, the MIN 1335 may transmit sorted results1345A-1345P, including the index values computed by the plurality ofmultipliers 1305A-1305P, to the VAR 1340.

Thus, the MIN 1335 and the MIN 1355 may provide a bi-directionalconnectivity (e.g., one direction for read and another direction forwrite) on first P ports (or k{circumflex over ( )}2 ports in case of nosparsity in an input sub-feature map) of a Benes network (total numberof ports may be (2k−1){circumflex over ( )}2), where the number of Pports may be based on a percentage of non-zero values in the inputsub-feature map. For example, if there are 60% non-zero values in aninput sub-feature map, and the kernel matrix is 3×3, then the P portsmay be chosen to be 6. In some embodiments, the P ports may be chosenindependent of the average sparsity and may be based on various hardwareconsiderations.

The VAR 1340 is initialized with a portion from an accumulator buffer1350 similar to the accumulators 1025A-1025P. The accumulator buffer1350 is similar to the register bank 1030. However, the accumulatorbuffer 1350 does not implement a shifting operation. Similar to theregister bank 1030, the accumulator buffer 1350 may be of a size of theoutput sub-feature map, as discussed above with respect to the registerbank. For example, using the example of FIG. 11, the accumulator buffer1350 may be of size 7×7. Further, at the start of the first iteration,the accumulator buffer 1350 may be initialized with all zero values, asdiscussed above.

Additionally, at the start of the first iteration, a portion of theaccumulator buffer 1350 may be loaded into the VAR 1340. The size of theVAR 1340 may be computed using the formula: (2k−1)×(2k−1), where k isthe kernel size of the kernel matrix 1120. Since in the example of FIG.11, the kernel size, k, of the kernel matrix 1120 is 2, the size of theVAR 1340 is 3×3. Further, the 3×3 portion from the accumulator buffer1350 that is copied into the VAR 1340 may be determined based on thefollowing formula:

VAR=accumulator buffer(i:i+acc_length−1,j:j+acc_length−1)

In the formula above, the first term corresponds to the row numbers(e.g., row index value) of the accumulator buffer 1350 and the secondterm corresponds to the column numbers (e.g., column index value) of theaccumulator buffer. Further, in the formula above, “i” is the start ofthe row number and “j” is the start of the column number of thesub-feature map 1125A in the padded feature map 1115, and acc_length isthe size of the accumulator buffer 1350. For example, since the size ofthe accumulator buffer 1350 in the example above is 3×3, the acc_lengthis 3. Thus, for example, for processing the sub-feature map 1125A, “i”in the formula above is 1, “j” is 1, and acc_length is 3. Thus, at thestart of the first iteration, rows 1:3 and columns 1:3 of theaccumulator buffer 1350 may be loaded into the VAR 1340.

Upon receiving the sorted results 1345A-1345P and the computed indexvalues from the MIN 1335, the VAR 1340 stores the results in theappropriate index value of the portion copied from the accumulatorbuffer 1350. Upon storing the sorted results 1345A-1345P in theappropriate index values in the VAR 1340, the VAR may transmit thesorted results 1345A-1345P to the MIN 1355, which is structured similarto the MIN 1335. The MIN 1355 may send the sorted results 1345A-1345Pback to the accumulator 1325 for use during the next iteration.

Further, after completing all iterations of processing the sub-featuremap 1125A, the VAR 1340 may send the results stored therein back to theaccumulator buffer 1350. For example, if rows 1:3 and columns 1:3 werecopied from the accumulator buffer 1350 to the VAR 1340 at the start ofthe first iteration, at the end of the last iteration, the results fromthe VAR are loaded back into rows 1:3 and columns 1:3 of the accumulatorbuffer. Thus, the portion of the accumulator buffer 1350 that is loadedinto the VAR 1340 at the start of the first iteration is replaced by theresults from the VAR 1340 at the end of the last iteration.

In addition to sending the product values 1320A-1320P to the accumulator1325, those product values are also input into an adder 1360. Theresults from the adder 1360 are stored within a special accumulator1365. The adder 1360 and the special accumulator 1365 are analogous tothe adder 1035 and the special accumulator 1040, respectively.

ReLU and pooling operations 1370 are performed, as discussed above, onthe values stored in the accumulator buffer 1350 after the lastiteration to obtain an output sub-feature map. The results of the ReLUand pooling operations 1370 may optionally be compressed in acompression block 1375, as discussed above. The compressed results maybe sent back to the DRAM 215 via the associated memory storage memorycluster or to another sparse tensor compute unit.

Referring now to FIG. 14, an example of a sparse tensor compute unit1400 is shown, in accordance with some embodiments of the presentdisclosure. The sparse tensor compute unit 1400 is substantially similarto the sparse tensor compute unit 1300. For example, similar to thesparse tensor compute unit 1300, the sparse tensor compute unit 1400includes a plurality of multipliers 1405A-1405P that receive data values1410A-1410P and weight values 1415A-1415P to generate product values1420A-1420P. Each of the plurality of multipliers 1405A-1405P may alsoreceive the index values of the data values 1410A-1410P and the weightvalues 1415A-1415P to compute the index values of the product values1420A-1420P, as explained above.

The sparse tensor compute unit 1300 is configured to receive one uniqueweight value at a time. In other words, the same weight value istransmitted to each of the plurality of multipliers 1305A-1305P in thesparse tensor compute unit 1300 in each iteration. However, the sparsetensor compute unit 1400 is configured to process more than one uniqueweight values in each iteration. In some embodiments, more than oneunique weight values may need to be processed at a time. In such cases,multiple weight values may be broadcast to the plurality of multipliers1405A-1405P at a time. For example, a first group of the plurality ofmultipliers 1405A-1405P may receive a first weight value, a second groupof the plurality of multipliers may receive a second weight value, andso on. For example, when two weight values are used in each iteration, afirst weight value may be transmitted to Q ones of the plurality ofmultipliers 1405A-1405P (where Q is the number of non-zeroes in aparticular sub-feature map) and a second weight value may be transmittedto (P−Q) number of the plurality of multipliers. The number of uniqueweight values that may be used in each iteration may be computed usingthe following formula: max(1, ceiling (P−Q)) where Q is the number ofnon-zero values in a particular sub-feature map and P is the number ofthe plurality of multipliers 1405A-1405P that are engaged.

The product values 1420A-1420P and the computed index values may betransmitted to a MIN 1425. Specifically, the MIN 1425 may be a writearbiter MIN having “P” number of first in first out buffers and P x Pcontrol states. If more than one of the product values 1420A-1420P havethe same computed index value in the output sub-feature map, the MIN1425 may send one of the product values going to that index value, whilestore the remaining product values in the buffers. The MIN 1425 thensends the products values 1420A-1420P and the computed index values toan accumulator 1430. The accumulator 1430 is similar to the accumulator1425. Further, MIN 1435, VAR 1440, MIN 1445, adder 1450, specialaccumulator 1460, accumulator buffer 1465, ReLU and pooling operations1470, and compression block 1475 are configured similar to the MIN 1335,the VAR 1340, the MIN 1355, the adder 1360, the special accumulator1365, the accumulator buffer 1350, the ReLU and pooling operations 1370,and the compression block 1375, and therefore, are not described againfor conciseness of expression. Each of these elements process thesub-feature map 1125A as described above in FIG. 14.

Turning to FIG. 15, an example of a sparse tensor compute unit 1500 isshown, in accordance with some embodiments of the present disclosure.The sparse tensor compute unit 1500 is substantially similar to thesparse tensor compute unit 1400. For example, similar to the sparsetensor compute unit 1400, the sparse tensor compute unit 1500 includes aplurality of multipliers 1505A-1505P that receive data values1510A-1510P and weight values 1515A-1515P to generate product values1520A-1520P, and compute the index values of those product values. Alsosimilar to the sparse tensor compute unit 1400, the sparse tensorcompute unit 1500 is configured to receive multiple weight values ineach iteration. The product values 1520A-1520P and the computed indexvalues may be transmitted to a MIN 1525, which may be structured similarto the MIN 1425. The MIN 1525 transmits values 1530A-1530P to anaddition block 1535. The addition block 1535 may perform the followingaddition operation:

S 1 = b 1 + b 2 S 2 = b 2 + b 3 S 3 = b 3 + b 4 … SP = bP + b 1

The added values (S1, S2, . . . SP) may be considered speculativecomputations. If two weight values are sent to the sparse tensor computeunit 1500, the two product values resulting from those two weight valuesmay need to be added together before accumulation. The added values (S1,S2, . . . , SP) may be transmitted as values 1540A-1540P to a selectunit 1545. The select unit 1545 may be a group of multiplexers that areconfigured to perform the following operations:

R 1 = MUX(b 1, SP, S 1) R 2 = MUX(b 2, S 1, S 2)R 3 = MUX(b 3, S 2, S 3) … RP = MUX(bP, S(P − 1), SP)

In some embodiments, the control for the selection of the multiplexersabove may be based on the output index value for each product value. Iftwo adjacent product values share the same output index value, then thesum value of the adjacent product values may be selected.

Although the select unit 1545 has been described as having a group ofmultiplexers, in other embodiments, other elements that are configuredto select one input from a plurality of inputs based on a selectioncriteria may be used.

Results 1550A-1550P (e.g., R1, R2, . . . , RP) may be transmitted to anaccumulator 1555. The accumulator 1555 is similar to the accumulator1430 and configured to perform the following addition operations:

a 1 = R 1 + a 2 a 2 = R 2 + a 3 a 3 = R 3 + a 4 … aP = RP + a 1

The results from the accumulator 1555 may be sent to a MIN 1560, whichis similar to the MIN 1435. Further, the results from the accumulator1555 may be processed in a VAR 1565, accumulator buffer 1570, MIN 1575,adder 1580, special accumulator 1585, ReLU and pooling operations 1590,and compression block 1595 similar to that in the VAR 1435, theaccumulator buffer 1465, the MIN 1445, the adder 1450, the specialaccumulator 1460, the ReLU and pooling operations 1470, and thecompression block 1475, and therefore, not explained again.

Turning to FIG. 16, an example flowchart outlining operations of aprocess 1600 is shown, in accordance with some embodiments of thepresent disclosure. The process 1600 may include other or additionaloperations depending upon the embodiment. The process 1600 may beimplemented in the sparse tensor compute unit 1300, the sparse tensorcompute unit 1400, or the sparse tensor compute unit 1500. In someembodiments, the sparse tensor compute unit implementing the process1600 may include a controller that performs the process 1600. Thecontroller may be associated with a memory for storing computer-readableinstructions, which may be executed by a processor associated with thecontroller. In other embodiments, the scheduling engine 225 may beconfigured to perform the process 1600. The process 1600 may beimplemented during a standard convolution operation in a CNN. Theprocess 1600 may also be implemented in a fully connected 1×1convolution operation in which a 1×1 weight matrix is applied.

Upon starting at operation 1605, the accumulator buffer (e.g., theaccumulator buffer 1350, the accumulator buffer 1465, the accumulatorbuffer 1570) of the sparse tensor compute unit (e.g., the sparse tensorcompute unit 1300, the sparse tensor compute unit 1400, or the sparsetensor compute unit 1500) that is implementing the process 1600 isinitialized at operation 1610. The operation 1610 is similar to theoperation 1210. As indicated above, at the start of the first iteration,the accumulator buffer may be initialized with all zero values. Further,upon initializing the accumulator buffer at the operation 1610, the VAR(e.g., the VAR 1340, the VAR 1440, the VAR 1565) of the sparse tensorcompute unit implementing the process 1600 is initialized at operation1615 such that a portion of the accumulator buffer is loaded into theVAR.

At operation 1620, data values from the sub-feature map (e.g., thesub-feature map 1125A) are input into the plurality of multipliers(e.g., the plurality of multipliers 1305A-1305P, the plurality ofmultipliers 1405A-1405P, the plurality of multipliers 1505A-1505P) ofthe sparse tensor compute unit implementing the process 1600, asdiscussed above. At the operation 1620, the index values correspondingto the data values may also be input into the plurality of multipliers.At operation 1625, one or more weight values (e.g., from the kernelmatrix 1220) are transmitted to each of the plurality of multipliers(e.g., the plurality of multipliers 1305A-1305P, the plurality ofmultipliers 1405A-1405P, the plurality of multipliers 1505A-1505P) ofthe sparse tensor compute unit implementing the process 1600, asdiscussed above. The index values of those weight values may also beinput into those multipliers. The number of weight values transmitted toeach of the plurality of multipliers may be dependent upon the type ofconvolution being performed. For example, in some embodiments, twounique weight values may be transmitted to each of the plurality ofmultipliers in a standard convolution operation, while a single weightvalue may be transmitted to each of those multipliers in a fullyconnected 1×1 convolution operation. In some embodiments, the operations1610/1615 and 1620/1625 may occur in parallel.

Each of the plurality of multipliers (e.g., the plurality of multipliers1305A-1305P, the plurality of multipliers 1405A-1405P, the plurality ofmultipliers 1505A-1505P) computes a product between the data value ofthe sub-feature map received therein and the weight value(s) atoperation 1630 to obtain a product value (e.g., the product values1320A-1320P, the product values 1420A-1420P, the product values1520A-1520P). The product values may be represented as a product matrix.At operation 1635, the plurality of multipliers (e.g., the plurality ofmultipliers 1305A-1305P, the plurality of multipliers 1405A-1405P, theplurality of multipliers 1505A-1505P) also compute the index values forthose product values, as discussed above.

At operation 1640, if two product values share the same computed index,the MIN (e.g., the MIN 1335, the MIN 1435, the MIN 1460) of the sparsetensor compute unit implementing the process 1600 sorts the productvalues such that the product values sharing the same index value areadjacent to each other in the VAR (e.g., the VAR 1340, the VAR 1440, theVAR 1565) of the sparse tensor compute unit implementing the process1600. The sorted results from the MIN are stored in the VAR at operation1645. Then, at operation 1650, it is determined whether all theiterations have been completed. If additional iterations remain, theprocess 1600 loops back to the operation 1610 to continue processing thenext iteration. On the other hand, if at the operation 1650, it isdetermined that all the iterations have completed, at operation 1655,the values in the VAR (e.g., the VAR 1340, the VAR 1440, the VAR 1565)obtained at the operation 1645 after completion of the last iterationare loaded back into the accumulator buffer (e.g., the accumulatorbuffer 1350, the accumulator buffer 1465, the accumulator buffer 1570).At operation 1660, ReLU and pooling operations are performed on thevalues in the accumulator buffer to obtain an output sub-feature map.Optionally, at operation 1665, the output sub-feature map may becompressed. The process 1600 then ends at operation 1670 by sending theoutput sub-feature map to the associated sparse tensor memory cluster400, 500, or to another sparse tensor compute unit.

In some embodiments, the output sub-feature map may again be sub-dividedinto additional sub-feature maps, as discussed above in FIGS. 3A and 3B,based on sparsity to equalize the number of zeroes and non-zeroes ineach of the additional sub-feature maps, and to equalize the workload ineach of the sparse tensor compute units of the sparse tensor computecluster 230. A similar rebalancing of the output sub-feature maps may beperformed in the process 1200.

In some embodiments, where depth separable convolutions are implemented(e.g., where the feature map has multiple channels), the result fromeach channel may be stored as an intermediate output sub-feature map.For example, the sub-feature maps of each channel may perform operations1605-1665, and the result of those operations may be stored as anintermediate output sub-feature map. In some embodiments, ReLU andpooling operations may not be performed on the intermediate outputsub-feature maps. Thus, depending upon the number of channels, multipleintermediate output sub-feature maps may be obtained. For example, forthree channels, three intermediate output sub-feature maps may beobtained. Each of the intermediate output sub-feature maps may then becombined by applying a 1×1 filter. The ReLU and pooling operations ofthe operation 1660 are then performed on the combined intermediateoutput sub-feature maps to obtain the output sub-feature map.

Turning now to FIG. 17, an example flowchart outlining operations of aprocess 1700 is shown, in accordance with some embodiments of thepresent disclosure. The process 1700 may include additional or otheroperations depending upon the particular embodiment. The process 1700may be implemented by the accelerator 200. In some embodiments, theprocess 1700 may be implemented by the scheduling engine 225 of theaccelerator 200. In other embodiment, the process 1700 may be implantedby another component of the accelerator 200. The process 1700 starts atoperation 1705 when one or more machine learning operations are to beperformed on a new input data (e.g., the input image 205) in a machinelearning application. For example, when an image classification is to bedone to identify one or more features in the input image, the process1700 may be implemented. Thus, at operation 1710, the input image or thefeature maps associated with the input image may be input into theaccelerator 200.

The input image may be represented by an array of pixels based upon thesize, height, and color scheme of the input image. Simply as an exampleand without intending to be limiting in any way, the process 1700 isexplained below with respect to an input image of size 1024×1024×3.Thus, the input image is 1024 pixel wide and 1024 pixel high, and has 3color channels. The input image may, thus, be treated as a combinationof three feature maps, with one feature map for each color channel andeach feature map having a size of 1024×1024×1. In some embodiments, theinput image may be input into the accelerator 200, and the acceleratormay derive the feature maps from the input image. In other embodiments,the conversion of the input image into the feature maps may occuroutside the accelerator 200 (by another component on the host device(e.g., the host device 105) with which the accelerator 200 isassociated), and the feature maps may be input into the acceleratorinstead of the input image. Further, the input image or the feature mapsof the input image (whichever is input into the accelerator 200) may bestored within the DRAM 215 of the accelerator.

At operation 1715, the accelerator 200 receives a weight matrix (e.g.,the weight matrix 220). Although the operation 1715 has been describedas occurring after the operation 1710 in which the input image or thefeature maps of the input image are received by the accelerator 200, insome embodiments, the operation 1715 may occur before the operation1710. In some embodiments, the operations 1710 and 1715 may occursimultaneously or substantially simultaneously.

In some embodiments, the accelerator 200 may be configured for a neuralnetwork having a plurality of layers. For example, in some embodiments,the accelerator 200 may be configured for a convolutional neural networkhaving a plurality of convolution layers. In some embodiments, each ofthe plurality of convolution layers may have a specific weight matrixthat may be applied to the feature map of that layer. Thus, in suchembodiments, the accelerator 200 may receive multiple weight matrices,with one weight matrix configured for one layer. Each weight matrix mayinclude weight values that may be applied to the feature maps.Specifically, multiplication, addition, and other operations may beperformed between corresponding weight values and the data values of theinput feature map. In some embodiments, the weight matrix may also becompressed. In some embodiments, the weight matrix may at leasttemporarily be stored within the DRAM 215.

In some embodiments, the weight matrices, before or after compression,may also be reordered, and a static sparsity analysis performed on theweight matrices. The static sparsity analysis may be performed before orafter the compression of the weight matrices. In some embodiments, thestatic sparsity analysis may be performed by the accelerator 200, whilein other embodiments, the static sparsity analysis may be performedoutside the accelerator. The static sparsity analysis identifies theweight values in a weight matrix that are zero. If a particular weightvalue is zero, the multiplication with that zero weight value is alsozero. Thus, instead of allocating resources to multiply zero weightvalues with values in the feature maps, the results of thosemultiplication operations may directly be encoded as zero. Further,since the weight values that are zero in the weight matrix are knownbeforehand and do not change from one feature map to another, thesparsity in a weight matrix is static.

Thus, the weight matrices may be compressed and reordered, and a staticsparsity analysis may be performed to identify zero weight values andobtain an input weight tensor from each weight matrix. Each of the inputweight values may be stored within the DRAM 215, and assigned forcomputation with an input feature map based on a load and store schedulestored within the accelerator 200.

At operation 1720, each of the input feature maps of the input image arecompressed, as discussed above. Although the operation 1720 is describedafter the operation 1715, in some embodiments, the operation 1720 mayoccur any time after the feature maps of the input image are available.Each of the feature maps may be compressed by recursively partitioningeach of the feature maps into portions of smaller cell sizes, until adesired compression criteria is reached. The desired compressioncriteria may be based on the number of compression levels or thethreshold minimum cell size, as discussed in greater detail in the U.S.application Ser. No. 16/726,084 mentioned above. The compressionoperations discussed throughout this disclosure may be performed asdiscussed in the U.S. application Ser. No. 16/726,084.

At operation 1725, each input feature map of the input image is dividedinto a plurality of sub-feature maps, as discussed in FIGS. 3A and 3Babove. By dividing an input feature map into a plurality of sub-featuremaps, each of the plurality of sub-feature maps may be processed inparallel. Further, the input feature map may be divided into theplurality of sub-feature maps, such that each of the plurality ofsub-feature maps has the same or substantially similar sparsity. In someembodiments, the sparsity in each of the plurality of sub-feature mapssatisfies a predetermined threshold (e.g., the predetermined percentagedifference threshold discussed above). Thus, each of the plurality ofsub-feature maps may have the same or similar number of zero valuesand/or non-zero values. Further, by dividing the input feature map intothe plurality of sub-feature maps, the dynamic sparsity in the inputfeature map may be converted into static or close to static sparsity ineach of the plurality of sub-feature maps.

Each of the plurality of sub-feature maps may be assigned to a sparsetensor compute unit based on the closeness of the where they are storedcurrently to the compute units (e.g., the sparse tensor compute units1000, 1300, 1400, 1500) at operation 1730. At operation 1735, each ofthe plurality of sub-feature maps is stored in the sparse tensor featuremap memory unit that is associated with the sparse tensor compute unitthat is to process a particular one of the plurality of sub-featuremaps. In some operations, a time folding operation may be performed if aparticular sub-feature map is larger than the storage capacity of thesparse tensor feature map memory unit in which that sub-feature map isto be stored. In some embodiments, a time folding operation may beconsidered similar to the sub-feature partitioning described above inFIGS. 3A and 3B. In a time folding operation, the partitioning is basedon the feature map size. For example, if the maximum size of an inputimage that is supported is 1024×1024 and if we have an input image thatis 2048×2048 in size, then the large input image may be divided intofour input feature maps, each input feature map having a size of1024×1024.

At the operation 1735, the weight values to be applied to thatparticular one of the plurality of sub-feature maps are also transferredto the sparse tensor weight memory associated with the sparse tensorcompute unit. When the sparse tensor compute unit becomes available, thesparse tensor feature map memory unit may transfer the particularsub-feature map to the sparse tensor compute unit. The sparse tensorweight memory may also transfer the weight matrix stored therein to thesparse tensor compute unit.

Each of the sparse tensor compute units processes the receivedsub-feature map to generate an output sub-feature map, at operation 1740and as discussed above with respect to FIGS. 10-16. Each of the sparsetensor compute units may send their output sub-feature map back to theDRAM 215 or to another sparse tensor compute unit. In some embodiments,one or more output sub-feature maps may again be subdivided according toFIGS. 3A and 3B, and undergo further processing in a sparse tensorcompute unit. Once all output sub-feature maps have completedprocessing, those output sub-feature maps may be combined to generatethe output feature map at operation 1745. The process 1700 ends atoperation 1750.

In some embodiments, at the output of each sparse tensor compute unit,the number of non-zeros in the output sub-feature map may be measured.As the output sub-feature maps are generated, it may be possible toredirect the output sub-feature map of one sparse tensor compute unit toanother sparse tensor compute unit for balancing sparsity. However, insome embodiments, it may be easier and more efficient to balance thesparsity in output sub-feature maps after all the sub-feature maps areproduced and stored in the sparse tensor feature map memories whilesupplying to the sparse tensor compute unit for the next convolutionlayer. For balancing sparsity in such a way, another network may be usedto provide connectivity between sparse tensor feature map memories andsparse tensor compute units. When processing depth-wise separableconvolutions, point-wise or 1×1 convolutions may be performed usingmultiple intermediate feature maps. These intermediate feature maps maynot be sparse as they just go through convolution only and not ReLU(activation). Also each intermediate feature map output may need to bestored back in DRAM. The sparsity arises at the final output ofpoint-wise or 1×1 convolutions as these include ReLU(activation).

Turning now to FIGS. 18A-18D, an example of processing an input featuremap 1800 in the sparse tensor compute unit 1300 is shown, in accordancewith some embodiments of the present disclosure. It is to be understoodthat the input feature map 1800 is simply an example and is not intendedto be limiting in any way. The present disclosure may be used to processinput feature maps of any size and having any data values therein. Theinput feature map 1800 may be generated from the input image 205. Asshown in FIG. 18A in particular, the input feature map 1800 may includea plurality of cells, each cell being formed at the intersection of arow (that extends in the X-direction 1105; see FIG. 11) and a column(that extends in the Y-direction 1110; see FIG. 11). Each of theplurality of cells in the input feature map 1800 includes a data valuethat is to be processed in the sparse tensor compute unit 1300.

Although not shown in FIG. 18A, in some embodiments, the input featuremap 1800 may be padded by zeroes on all sides similar to the inputfeature map 1100. Further, in some embodiments, sub-feature maps may becreated from the input feature map 1800 (or from the padded feature mapwhen padding is used). In some embodiments, the input feature map 1800may be divided into sub-feature maps in the partitioning block 210 orother component of the accelerator 200, as discussed above in FIGS. 3Aand 3B. For example and as shown in FIG. 18A, the input feature map 1800may be divided into a plurality of sub-feature maps 1805A-1805P. Each ofthe plurality of sub-feature maps 1805A-1805P may be processed in thesparse tensor compute unit 1300 in series (or in parallel in multipleinstances of the sparse tensor compute unit 1300). In some embodiments,the plurality of sub-feature maps 1805A-1805P may be processed in anorder shown by arrows 1810. For example, the sub-feature map 1805A maybe processed first followed by the sub-feature maps 1805B-1805D. Then,the sub-feature maps 1805E-1805H may be processed, followed by thesub-feature maps 18051-1805L, and so on. Further, within each of theplurality of sub-feature maps 1805A-1805P, the data values may beprocessed in an order shown by arrow 1815. The order in which theplurality of sub-feature maps 1805A-1805P are processed and/or the orderin which the data values of each of the plurality of sub-feature mapsare processed may vary in other embodiments. Additionally, in someembodiments, only non-zero data values may be input into the sparsetensor compute unit 1300.

Although each of the plurality of sub-feature maps 1805A-1805P includestwo rows and two columns and those sub-feature maps are all of the samesize, it is to be understood that in other embodiments, thosesub-feature maps may be of varying sizes, may include varying numbers ofrows and columns, and the data values within each of the plurality ofsub-feature maps 1805A-1805P may vary. Further, depending upon thenumber of the plurality of multipliers 1305A-1305P in the sparse tensorcompute unit 1300, multiple data values of each of the plurality ofsub-feature maps 1805A-1805P may be processed in parallel. For purposesof explanation, FIGS. 18B-18D are explained with respect to thesub-feature map 1805A. However, the other ones of the plurality ofsub-feature maps 1805B-1805P may be similarly processed. Further, FIGS.18A-18D are described assuming that the number of the plurality ofmultipliers 1305A-1305P is three. Since the sub-feature map 1805Aincludes four non-zero data values, each iteration may include tworounds. In the first round, three data values of the sub-feature map1805A may be processed and in the second round, the last data value ofthat sub-feature map may be processed. Since the data values areprocessed in the order shown by the arrow 1815, in the first round, thedata values having the indices (0, 0), (0, 1), and (1, 0) may beprocessed, while in the second round the data value having the index(1, 1) may be processed.

The data values of the sub-feature map 1805A may be combined with weightvalues of a kernel matrix 1820. It is to be understood that the kernelmatrix 1820 is simply an example and is not intended to be limiting inany way. The kernel matrix 1820 may assume other sizes (e.g., the numberof rows and number of columns may vary from that shown) and the valueswithin the kernel matrix may also vary. Further, although each of theplurality of sub-feature maps 1805A-1805P has the same number of rowsand the columns as the kernel matrix 1820, in some embodiments, one ormore of the plurality of sub-feature maps may have varying number ofrows and/or columns than the kernel matrix. The weight values in thekernel matrix 1820 may be applied in a designated order as shown byarrow 1825. Further, since the sparse tensor compute unit 1300 isconfigured for one unique weight value in each iteration, a singleweight value is broadcast to each of the plurality of multipliers1305A-1305P in each iteration.

Thus, to process the sub-feature map 1805A, the data values from thatsub-feature map may be input into the plurality of multipliers1305A-1305P. Assuming three multipliers, and as shown in FIG. 18B, thefirst three data values may be input into the plurality of multipliers1305A-1305P. Specifically, the data value “3” may be input into themultipliers 1305A, the data value “9” may be input into the multiplier1305B, and the data value “6” may be input into the multiplier 1305C.Further, each of the multipliers 1305A-1305C may receive the firstweight value of “−6,” as shown in FIG. 18B and compute a product betweentheir respective data values and the first weight value. Each of themultipliers 1305A-1305C may also receive the index values of the datavalues that each of those multipliers receive, as well as the indexvalue of the first weight value. Based on the received index values,each of those multipliers 1305A-1305C may compute the result indexvalues in the output sub-feature map where the results of themultiplication are to stored. In some embodiments, the multipliers1305A-1305C may compute the result index values using the formulaediscussed above in FIG. 13.

The product values from the multipliers 1305A-1305C and the result indexvalues may be sent to the accumulator 1325. The accumulator 1325 mayalso receive the addition results from the previous iterations from theMIN 1355, as shown in FIG. 18B. Specifically, the MIN 1355 may alsoreceive the result index values and read the values from the VAR 1340corresponding to the result index values, and send those values to theaccumulator 1325, as shown in FIG. 18B. The accumulator 1325 may thenadd the product values of the current iteration to the product values ofthe previous iterations. The accumulator 1325 may transmit the resultsof the addition, as well as the result index values to the MIN 1335. TheMIN 1335 may write the results of the addition to the VAR 1340 at theindex locations identified by the result index values. As indicatedabove, the VAR 1340 is initialized with a portion from the accumulatorbuffer 1350. Thus, upon writing the results of the additions to the VAR1340 at the index locations identified by the result index values, theVAR 1340 may have values as shown in an output map 1830, only a portionof which is shown in FIG. 18B.

In the second round of the first iteration, the remaining data value,“−5,” of the sub-feature map 1805A is input into the multiplier 1305A.The first weight value, “−6,” is also input into the multiplier 1305A,as shown in FIG. 18C. The index values of the data value and the firstweight value are also input into the multiplier 1305A. The product valueand the result index value are sent to the accumulator 1325, which addsthe product value to the product value from the previous iterations, andsends the result to the VAR 1340 via the MIN 1335, as shown in theoutput map 1830 of FIG. 18C.

Referring now to FIG. 18D, in the first round of the second iteration,the data values “3,” “9,” and “6” are input into the multipliers 1305A,1305B, and 1305C, respectively. Further, the second weight value “6” isinput into the multipliers 1305A-1305C, as shown in FIG. 18D.Additionally, the index values of the data values and the second weightvalue may be input into the multipliers 1305A-1305C. The data values andthe second weight value may be combined in the sparse tensor computeunit 1300, as discussed above with respect to the first round of thefirst iteration. The output of the first round of the second iterationmay be stored in the output map 1830, as shown in FIG. 18D. The secondround of the second iteration may be similar to the second round of thefirst iteration but with the second weight value instead of the firstweight value. Similarly, each of the third weight value, “−9,” and thefourth weight value, “3” may be processed in the third iteration and thefourth iteration, respectively. Each of the third iteration and thefourth iteration may have two rounds, similar to the first iteration andthe second iteration. The values in the output map 1830 at the end ofthe fourth iteration may form the output sub-feature map and may bestored within the accumulator buffer 1350.

Turning now to FIGS. 19A-19E, an example of processing an input featuremap 1900 in the sparse tensor compute unit 1500 is shown, in accordancewith some embodiments of the present disclosure. It is to be understoodthat the input feature map 1900 is simply an example and is not intendedto be limiting in any way. The present disclosure may be used to processinput feature maps of any size and having any data values therein. Theinput feature map 1900 may be generated from the input image 205. Asshown in FIG. 19A in particular, the input feature map 1900 may includea plurality of cells, each cell being formed at the intersection of arow (that extends in the X-direction 1105; see FIG. 11) and a column(that extends in the Y-direction 1110; see FIG. 11). Each of theplurality of cells in the input feature map 1900 includes a data valuethat is to be processed in the sparse tensor compute unit 1500.

Although not shown in FIG. 19A, in some embodiments, the input featuremap 1900 may be padded by zeroes on all sides similar to the inputfeature map 1100. Further, in some embodiments, sub-feature maps may becreated from the input feature map 1900 (or from the padded feature mapwhen padding is used). In some embodiments, the input feature map 1900may be divided into sub-feature maps in the partitioning block 210 orother component of the accelerator 200, as discussed above in FIGS. 3Aand 3B. For example and as shown in FIG. 19A, the input feature map 1900may be divided into a plurality of sub-feature maps 1905A-1905P. In someembodiments, at least some of the plurality of sub-feature maps1905A-1905P may be processed in the sparse tensor compute unit 1500 inseries (e.g., one after another). In other embodiments, at least some ofthe plurality of sub-feature maps 1905A-1905P may be processed inparallel in multiple instances of the sparse tensor compute unit 1500.In some embodiments, the plurality of sub-feature maps 1905A-1905P maybe processed in an order shown by arrows 1910. Further, within each ofthe plurality of sub-feature maps 1905A-1905P, the data values may beprocessed in an order shown by arrow 1915. The order in which theplurality of sub-feature maps 1905A-1905P are processed and/or the orderin which the data values of each of the plurality of sub-feature mapsare processed may vary in other embodiments. Additionally, in someembodiments, only non-zero data values may be input into the sparsetensor compute unit 1500.

Although each of the plurality of sub-feature maps 1905A-1905P includestwo rows and two columns and those sub-feature maps are all of the samesize, it is to be understood that in other embodiments, thosesub-feature maps may be of varying sizes, may include varying numbers ofrows and columns, and the data values within each of the plurality ofsub-feature maps 1905A-1905P may vary. Further, depending upon thenumber of the plurality of multipliers 1505A-1505P in the sparse tensorcompute unit 1500, multiple data values of each of the plurality ofsub-feature maps 1505A-1505P may be processed in parallel. For purposesof explanation, FIGS. 19B-19E are explained with respect to thesub-feature map 1905A. However, the other ones of the plurality ofsub-feature maps 1905B-1905P may be similarly processed. Further, FIGS.19A-19E are described assuming that the number of the plurality ofmultipliers 1505A-1505P is three. Since the sub-feature map 1905Aincludes only two non-zero data values, each iteration may include asingle round. It is to be understood that the number of rounds in eachiteration may vary from one sub-feature map to another sub-feature map.For example, the sub-feature map 1905B includes three non-zero datavalues and since three multipliers are used in the current example, eachiteration of processing that sub-feature map may still include a singleround. On the other hand, the sub-feature map 1905F includes fournon-zero data values. Thus, with three multipliers, each iteration ofprocessing the sub-feature map 9105F may include two rounds.

The data values of the sub-feature map 1905A may be combined with weightvalues of a kernel matrix 1920. It is to be understood that the kernelmatrix 1920 is simply an example and is not intended to be limiting inany way. The kernel matrix 1920 may assume other sizes (e.g., the numberof rows and number of columns may vary from that shown) and the valueswithin the kernel matrix may also vary. Further, although each of theplurality of sub-feature maps 1905A-1905P has the same number of rowsand the columns as the kernel matrix 1920, in some embodiments, one ormore of the plurality of sub-feature maps may have varying number ofrows and/or columns than the kernel matrix. The weight values in thekernel matrix 1920 may be applied in a designated order as shown byarrow 1925. Further, as indicated above, the sparse tensor compute unit1500 is configured to process multiple weight values in each iteration.Thus, depending upon the number of non-zero data values and the numberof multipliers, multiple weight values may be broadcast in eachiteration. The example of FIGS. 19A-19E is explained assuming two uniqueweight values may be broadcast in each iteration.

For example and as shown in FIG. 19B, the sub-feature map 1905A includestwo non-zero values only. Thus, only the first two of the threemultipliers need to be engaged in the first iteration to process the twonon-zero data values. However, to increase performance and utilizeresources at full capacity, a second weight value and the first non-zerodata value may be broadcast to the third multiplier. For example, thedata values “9” and “−5” of the sub-feature map 1905A may be input intothe multiplier 1505A and 1505B, respectively. Further, the first weightvalue “−6” may be input into the multipliers 1505A and 1505B. Since wehave three multipliers, the first non-zero data value in the order shownby the arrow 1915 is “9,” which is input into the multiplier 1505C.Further, the second weight value “6” is input into the multiplier 1505Cto start the second iteration. Thus, the second iteration overlaps withthe first iteration.

Each of the multipliers 1505A-1505C may also receive the index values ofthe data values that each of those multipliers receive, as shown in FIG.19B. Further, each of the multipliers 1505A and 1505B receives the indexvalue of the first weight value and the multiplier 1505C receives theindex value of the second weight value. Based on the received indexvalues of the data values, the first weight value, and the second datavalue, each of those multipliers 1505A-1505C may compute the resultindex values in the output sub-feature map where the results of themultiplication are to be stored, as discussed above. The product resultsas well as the result index values may be transmitted to the MIN 1525.Since the result index values computed in the multipliers 1505A-1505Care different from one another, the MIN 1525 simply passes the productresults and the result index values to the addition block 1535.

The addition block 1535 may perform the following additions:

s1=b1+b3

s2=b2+b1

s3=b3+b2

In the formulae above, b1, b2, and b3 are the outputs from the MIN 1525and correspond to the product results generated by the multipliers1505A, 1505B, and 1505C, respectively. As further shown in FIG. 19B, thesums from the addition block 1535 and the result index values may beinput into the select unit 1545, which includes a plurality ofmultiplexers. In some embodiments, to process results from threemultipliers, the select unit 1545 may include three multiplexers.

The results from the multiplexers of the select unit 1545 and the resultindex values may be sent to the accumulator 1555, which adds the s1, s2,and s3 values respectively to those values from the previous iteration,as discussed above and shown in FIG. 19B. Specifically, the result indexvalues may be input into the MIN 1575, which may read valuescorresponding to those index values from the VAR 1565 and transmit thoseread values to the accumulator 1555. The results from the accumulatorand the result index values may be transmitted to the MIN 1560 andwritten into the VAR 1565.

FIG. 19C shows the second iteration (or rather the second half of thesecond iteration), which overlaps with the second iteration. Asdiscussed above, the second weight value and the data value “9” areprocessed during the first iteration (or the first half of the seconditeration). In the second half of the second iteration, the remainingnon-zero data value “−5” is input into the multipliers 1505A along withthe second weight value. Since there are three multipliers, theremaining two non-zero data values are input into the multipliers 1505Band 1505C along with the third weight value, “−9.” Thus, the second halfof the second iteration and the third iteration occur in parallel. Theprocessing of the second iteration is similar to that of the firstiteration, and therefore, not described again. Similarly, thesub-feature map 1905A may be processed with the remaining weight values.

FIG. 19D shows an example of the sub-feature map 1905C having a singlenon-zero data value. With a single non-zero data value and threemultipliers, in some embodiments, three weight values may be applied ina single iteration. For example, in the first iteration, the non-zerodata value “−2” may be input into the multiplier 1905A along with thefirst weight value “−6.” With two remaining multipliers, the non-zerodata value “−2” may also be input into the multiplier 1905B along withthe second weight value “6” and into the multiplier 1905C along with thethird weight value “−9.” However, if the sparse tensor compute unit 1900is configured to limit the number of unique weight values in aparticular iteration to two, only the first and the second data weightvalues may be processed in the first iteration. In such a case, themultiplier 1905C may not receive the non-zero data value “−2” with thethird weight value. Rather, the multiplier 1905C may sit idle, as shownin FIG. 19D. The processing in the first iteration using the multipliers1905A and 1905B may proceed similar to what is described above.

FIG. 19E shows an example in which the result index values of oneiteration and the overlapping half of the next iteration overlap. Forexample, the sub-feature map 1905J includes two non-zero values, whichmay be processed similar to what is describe above for the sub-featuremap 1905A. However, the result index value of (5, 2) computed for theproduct of the data value “9” and the first weight value of “−6” is sameas the result index value of (5, 2) computed for the product of the datavalue “−9” and the second weight value of “6.” In such a case, a mergeoperation may be performed, as discussed in FIGS. 20A and 20B below.

The example of FIGS. 19A-19E may also be used to process the inputfeature map 1900 using the sparse tensor compute unit of FIG. 14. Thesparse tensor compute unit 1400 is intended to be used with two uniqueweight values. Thus, the example of FIGS. 19A-19E may also apply to thesparse tensor compute unit 1400. The sparse tensor compute unit 1400 issimilar to the sparse tensor compute unit 1500 with the exception of theaddition block 1535 and the select unit 1545. Since the MIN 1525 issimilar to the MIN 1425, the data from the MIN 1425 may be transmitteddirectly to the accumulator 1430 in the sparse tensor compute unit 1400instead of the MIN 1525 transmitting the data to the addition block 1535in the sparse tensor compute unit 1500.

The MIN 1425 and the MIN 1525 may both be used to avoid a collision whentwo product values have the same result index value. When such acollision occurs, the MIN 1425 and the MIN 1525 apply a mergingoperation, discussed in FIGS. 20A and 20B below.

Turning now to FIGS. 20A and 20B, an example of a merging operation isexplained, in accordance with some embodiments of the presentdisclosure. The merging operation is described with respect to thesparse tensor compute unit 1400, but the merging operation may similarlybe implemented in the sparse tensor compute unit 1500. As discussedabove with respect to the sub-feature map 1905J and FIG. 19E, the resultindex value of (5, 2) computed for the product of the data value “9” andthe first weight value of “−6” is same as the result index value of (5,2) computed for the product of the data value “−9” and the second weightvalue of “6.” The merging operation resolves the collision in the resultindex value. The merging operation may be implemented by the MIN 1425 inthe sparse tensor compute unit 1400 (or the MIN 1525 in the sparsetensor compute unit 1500). In some embodiments, the MIN 1425 (and theMIN 1525) may include a FIFO (first in first out buffer) 2000 to resolvethe collision.

When the MIN 1425 (or the MIN 1525) receives the result index valuesfrom the plurality of multipliers 1405A-1405P, upon detecting thecollision, the MIN may transfer one of the product values and thecorresponding result index value to the FIFO 2000. In some embodiments,the product value and the corresponding result index value that istransferred to the FIFO 2000 may be based on the order in which theweight values are being processed. For example, in the example of thesub-feature map 1905J, the product value corresponding to the secondweight value may be transmitted to the FIFO 2000 instead of the productvalue corresponding to the first weight value. In some embodiments, theMIN 1425 (and the MIN 1525) may be programmed with which product valueto transfer to the FIFO 2000. If there are more than two collisions(e.g., more than two product values having the same result index value),all colliding product values may be sent to the FIFO 2000 except one.

Upon transferring the product value(s) to the FIFO 2000, the remainingproduct values and their corresponding result index values may betransmitted to the accumulator 1430 (in the sparse tensor compute unit1400) or to the addition block 1535 (in the sparse tensor compute unit1500). Thus, when three values are being processed in parallel in theplurality of multipliers 1405A-1405C and there is a collision betweentwo product values, one of the colliding product values is stored in theFIFO 2000, and only two product values are further processed. Theproduct value stored in the FIFO 2000 is processed in the nextround/iteration. Thus, as shown, in FIG. 20B, in the nextround/iteration of processing the sub-feature map 1905J, only two datavalues are input into the plurality of multipliers 1405A-1405P togenerate two product values. The two product values are sent to the MIN1425, and if there is no collision between the product values receivedby the MIN and the product value stored in the FIFO 2000, three productvalues are output by the MIN, as shown in FIG. 20B. The remainingdownstream processing then proceeds normally, as discussed above.

The various illustrative logical blocks and modules described inconnection with the embodiments disclosed herein can be implemented orperformed by a machine, such as a general purpose processor device, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. A control processor can synthesize a modelfor an FPGA. For example, the control processor can synthesize a modelfor logical programmable gates to implement a tensor array and/or apixel array. The control channel can synthesize a model to connect thetensor array and/or pixel array on an FPGA, a reconfigurable chip and/ordie, and/or the like. A general-purpose processor device can be amicroprocessor, but in the alternative, the processor device can be acontroller, microcontroller, or state machine, combinations of the same,or the like. A processor device can include electrical circuitryconfigured to process computer-executable instructions. In anotherembodiment, a processor device includes an FPGA or other programmabledevice that performs logic operations without processingcomputer-executable instructions. A processor device can also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Although described herein primarily with respect todigital technology, a processor device may also include primarily analogcomponents. For example, some or all of the algorithms described hereinmay be implemented in analog circuitry or mixed analog and digitalcircuitry. A computing environment can include any type of computersystem, including, but not limited to, a computer system based on amicroprocessor, a mainframe computer, a digital signal processor, aportable computing device, a device controller, or a computationalengine within an appliance, to name a few.

The elements of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processordevice, or in a combination of the two. A software module can reside inRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, a CD-ROM, or any other form of anon-transitory computer-readable storage medium. An exemplary storagemedium can be coupled to the processor device such that the processordevice can read information from, and write information to, the storagemedium. In the alternative, the storage medium can be integral to theprocessor device. The processor device and the storage medium can residein an ASIC. The ASIC can reside in a user terminal. In the alternative,the processor device and the storage medium can reside as discretecomponents in a user terminal.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable,” to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to inventions containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.).

In those instances, where a convention analogous to “at least one of A,B, or C, etc.” is used, in general such a construction is intended inthe sense one having skill in the art would understand the convention(e.g., “a system having at least one of A, B, or C” would include butnot be limited to systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.). It will be further understood by those within the artthat virtually any disjunctive word and/or phrase presenting two or morealternative terms, whether in the description, claims, or drawings,should be understood to contemplate the possibilities of including oneof the terms, either of the terms, or both terms. For example, thephrase “A or B” will be understood to include the possibilities of “A”or “B” or “A and B.” Further, unless otherwise noted, the use of thewords “approximate,” “about,” “around,” “substantially,” etc., mean plusor minus ten percent.

The foregoing description of illustrative embodiments has been presentedfor purposes of illustration and of description. It is not intended tobe exhaustive or limiting with respect to the precise form disclosed,and modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the disclosed embodiments.It is intended that the scope of the invention be defined by the claimsappended hereto and their equivalents.

What is claimed is:
 1. An apparatus comprising: a tensor compute clustercomprising a plurality of tensor compute units to process a plurality ofsub-feature maps in a machine learning application; a tensor memorycluster comprising a plurality of tensor feature map memory units tostore the plurality of sub-feature maps; and circuitry to: partition aninput feature map into the plurality of sub-feature maps such thatsparsity in each of the plurality of sub-feature maps satisfies apredetermined threshold; and assign each of the plurality of sub-featuremaps to one of the plurality of tensor compute units and one of theplurality of tensor feature map memory units for processing in parallel.2. The apparatus of claim 1, wherein each of the plurality of tensorfeature map memory units is connected to one of the plurality of tensorcompute units, such that the circuitry sends a first sub-feature map ofthe plurality of sub-feature maps assigned to a first tensor computeunit of the plurality of tensor compute units to be stored in a firsttensor feature map memory unit of the plurality of tensor feature mapmemory units associated with the first tensor compute unit.
 3. Theapparatus of claim 1, wherein the tensor memory cluster furthercomprises a tensor weight memory for storing a kernel matrix.
 4. Theapparatus of claim 3, wherein the tensor weight memory comprises aplurality of buffers for storing the kernel matrix and sending thekernel matrix to the tensor compute cluster.
 5. The apparatus of claim1, wherein each of the plurality of tensor feature map memory unitscomprises a plurality of buffers, a write switch to write the pluralityof sub-feature maps in the plurality of buffers, and a read switch toread the plurality of sub-feature maps from the plurality of buffers. 6.The apparatus of claim 5, wherein the plurality of buffers comprises afirst buffer for storing data values of at least one of the plurality ofsub-feature maps and a second buffer for storing index valuescorresponding to the data values.
 7. The apparatus of claim 1, whereinto partition the input feature map into the plurality of sub-featuremaps, the circuitry: divides the input feature map into a predeterminednumber of initial partitions; and reorganizes at least some of theinitial partitions such that each of the initial partitions satisfiesthe predetermined threshold after the reorganization.
 8. The apparatusof claim 7, wherein the predetermined number of the initial partitionsis based upon a number of the plurality of tensor compute units.
 9. Theapparatus of claim 7, wherein the circuitry reorganizes at least some ofthe initial partitions by reallocating a row and/or a column to a firstinitial partition from a neighboring initial partition.
 10. Theapparatus of claim 1, wherein a first sub-feature map of the pluralityof sub-feature maps is of a different size than a second sub-feature mapof the plurality of sub-feature maps.
 11. The apparatus of claim 1,wherein each of the plurality of tensor compute units comprises aplurality of multipliers, and wherein each of the plurality ofmultipliers computes a product between a data value of a firstsub-feature map of the plurality of sub-feature maps and a weight valuefrom a kernel matrix.
 12. The apparatus of claim 1, wherein each of theplurality of tensor compute units processes one of the plurality ofsub-feature maps at a time to generate an output sub-feature map, andwherein the circuitry combines the output sub-feature map from each ofthe plurality of tensor compute units to obtain an output feature mapfor the input feature map.
 13. A method comprising: partitioning, by anaccelerator in a machine learning application, an input feature map intoa plurality of sub-feature maps, wherein sparsity in each of theplurality of sub-feature maps satisfies a predetermined threshold;assigning, by the accelerator, each of the plurality of sub-feature mapsto one of a plurality of tensor compute units of a tensor computecluster for processing each of the plurality of sub-feature maps inparallel; processing, by the accelerator, each of the plurality ofsub-feature maps in the assigned one of the plurality of tensor computeunits to generate an output sub-feature map from each of the pluralityof tensor compute units; and combining, by the accelerator, the outputsub-feature map from each of the plurality of tensor compute units togenerate an output feature map for the input feature map.
 14. The methodof claim 13, wherein partitioning the input feature map into theplurality of sub-feature map comprises: dividing the input feature mapinto a predetermined number of initial partitions, wherein thepredetermined number of the initial partitions is based upon a number ofthe plurality of tensor compute units; and reorganizing at least some ofthe initial partitions such that each of the initial partitionssatisfies the predetermined threshold after the reorganization.
 15. Themethod of claim 13, wherein when the predetermined threshold issatisfied, each of the plurality of sub-feature maps comprises asubstantially similar number or percentage of non-zero values.
 16. Themethod of claim 15, wherein at least some of the plurality ofsub-feature maps are of a different size than other ones of theplurality of sub-feature maps.
 17. A non-transitory computer-readablemedia having computer-readable instructions stored thereon that, whenexecuted by a processor associated with an accelerator in a machinelearning application, cause the processor to: partition an input featuremap into a plurality of sub-feature maps such that each of the pluralityof sub-feature maps comprises substantially similar sparsity; and assigneach of the plurality of sub-feature maps to a sparse tensor computecluster for processing each of the plurality of sub-feature maps inparallel.
 18. The non-transitory computer-readable media of claim 17,wherein the sparse tensor compute cluster comprises a plurality ofsparse tensor compute units, and the processor comprisescomputer-readable instructions to assign each of the plurality ofsub-feature maps to one of the plurality of sparse tensor compute units.19. The non-transitory computer-readable media of claim 18, wherein eachof the plurality of sparse tensor compute units is associated with oneof a plurality of sparse tensor feature map memory units of a sparsetensor memory cluster of the accelerator.
 20. An apparatus comprising:means for partitioning an input feature map into a plurality ofsub-feature maps based upon sparsity; and means for assigning each ofthe plurality of sub-feature maps to a tensor compute cluster forprocessing each of the plurality of sub-feature maps in parallel.